wehub/alirezarezvani--claude-skills
1
0
Fork 0
Code Issues Pull Requests Actions 3 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Release / Tag + GitHub Release (push) Waiting to run
Details
Deploy Documentation to Pages / build (push) Waiting to run
Details
Deploy Documentation to Pages / deploy (push) Blocked by required conditions
Details
Sync Codex Skills Symlinks / sync (push) Waiting to run
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:41:47 +08:00
commit a1fa97429b
4594 changed files with 736231 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
audit/engineering-agentic-2026-07/00-MASTER.md
+137
View File
@@ -0,0 +1,137 @@
# Master report — Engineering agentic-loop audit + agent-harness framework
**Audited:** 2026-07-03 · **Branch:** `claude/engineering-audit-agentic-loops-hv9x9m` ·
**Scope:** both engineering domain folders — `engineering/` (63 skills) and
`engineering-team/` (52 skills incl. sub-skills) — re-audited against the June 2026 baseline
AND scored on a new **agentic-readiness** rubric. Plus: a new `engineering/agent-harness`
skill that turns any of the repo's 18 domains into a bounded, self-verifying agent loop.
**Method:** (1) read the June `audit/newgen-2026-06/` reports; (2) two parallel deep-dive
agents re-read every SKILL.md, re-ran the June "Verify" criteria, and smoke-tested ~90
scripts; (3) one research agent web-verified the 2025–2026 agent-harness canon
([research-digest.md](research-digest.md)); (4) one explorer mapped the repo's existing
loop infrastructure so the new skill reuses rather than duplicates it.
---
## 1. The two questions
June asked: **does each skill earn its context window?** (trigger quality, wiring,
freshness). That audit drove a wave of fixes — REWRITEs, a phantom-path sweep, orphan-script
wiring, a 100-line ceiling.
This audit asks the next question: **can an agent pick a skill up with a goal and drive it
to a verified close?** — the gather→act→verify→repeat loop the harness literature converged
on. The rubric ([RUBRIC.md](RUBRIC.md)) scores six dimensions 0–2: goal intake (AR1), task
decomposition (AR2), deterministic execution (AR3), verification (AR4), loop discipline
(AR5), close-out (AR6).
---
## 2. Combined scorecard (115 skills across both folders)
| Class | engineering/ | engineering-team/ | Total | Meaning |
|---|---|---|---|---|
| **HARNESS-READY** (≥9, AR4≥1, AR5≥1) | 22 | 4 | **26** | An agent can loop this today |
| **LOOP-CAPABLE** (6–8) | 23 | 16 | **39** | One or two additions away |
| **TOOL-ONLY** (3–5) | 16 | 27 | **43** | Good tools, no loop spine |
| **PROSE-ONLY** (0–2) | 2 | 5 | **7** | Needs structural rebuild |
**Delta vs June** (35 non-KEEP verdicts across both folders): RESOLVED 16 ·
PARTIALLY-RESOLVED 10 · STILL-OPEN 9. The June wave landed most of its wiring and
correctness fixes; what remains is structural (deferred merges/dedupes) and the *new*
dimension this audit adds.
---
## 3. The single biggest finding: AR5 (loop discipline) is the repo-wide gap
The June wiring epidemic is largely cured — AR3 (deterministic execution) is now the median
strength. But **loop discipline is the weakest dimension in both folders.** Skills describe
"re-run until clean" with no iteration cap, or have no stop condition at all. Only the
v2.4+/Pocock/orchestrator generation (agenthub, autoresearch, chaos-engineering,
grill-with-docs, workflow-builder, playwright-pro/fix, the upgraded senior-* trio) carries
caps and escalation thresholds.
**Why it matters most for autonomous work:** a skill with great intake and tools but no
verification gate or stop condition is *more* dangerous in a loop, not less — it runs
confidently and forever, and (per the reward-hacking literature) may learn to game its own
checks. That is exactly why the AR class gate requires AR4≥1 **and** AR5≥1 before a skill
counts as harness-ready regardless of total score.
**Cheapest high-leverage fix:** a one-sentence loop-cap pattern —
*"max N fix-rerun cycles, then escalate to a human"* — ported across the ~15 skills sitting
at 7–8 points would roughly double the HARNESS-READY count. The pattern already exists
in-house (playwright-pro/fix, spec-driven-workflow, focused-fix's 3-strike rule).
Second gap: **AR1 (goal intake).** Most tool-rich skills accept any input silently. The
decision-engine "refuse without required inputs" pattern from the upgraded senior-* trio and
grill-me is the cheapest fix to propagate.
---
## 4. What this PR ships: the agent-harness framework
Rather than hand-fix 100 skills, this PR builds the **thin unifying layer** the explorer
found missing — each existing loop primitive (agenthub, autoresearch, tc-tracker,
workflow-builder, the fork-orchestrators) ships its own state dir, state machine, and eval
contract, with nothing that lets an agent pick up an arbitrary goal for an arbitrary domain
and drive it to a verified close. `engineering/agent-harness` is that layer:
- **`harness_manifest_builder.py`** scans a domain folder → a `manifest.v1` JSON inventory
(every skill, its tools, the exact `--help`/`--sample` checks that prove each tool works,
and static `agentic_signals` mapping to AR1/AR4/AR5/AR6). **18 domain manifests are
committed** under the skill's `assets/harnesses/` — the whole repo, machine-readable.
- **`goal_compiler.py`** turns a goal + manifest into a `plan.v1` task plan (deterministic
keyword scoring, no LLM call). **Refuses vague goals (exit 3)** with forcing questions and
**refuses no-match (exit 4)** with nearest candidates — the harness never runs on fuzz.
- **`loop_controller.py`** is the JSON-backed state machine: `init → next → record →
verify → close`. It **runs verification checks itself via subprocess** (no verification
theater), **caps attempts and iterations** (escalates instead of looping forever), and
**refuses to close** while any task is unverified and unwaived (exit 4, no force flag).
Plus `harness-runner` agent (stateless one-task-per-invocation executor), `/cs:harness`
command, a JSON schema, and 3 references citing the canon. Every design decision traces to a
source: verifier's law, SWE-agent's write-time feedback, Ralph fresh-context iteration,
Cognition's serialize-writers rule, Anthropic's long-running-agents harness, and this repo's
own tc-tracker / autoresearch locked-evaluator / loop-library stop-state taxonomy.
**Reuse, not reinvention.** The harness routes to agenthub for N-agent tournaments, to
autoresearch for metric optimization, and adopts tc-tracker's atomic-write + handoff schema
and autoresearch's "never modify the evaluator" invariant. See the reuse map in the skill's
`references/domain_harness_design.md`.
---
## 5. Per-domain reports
- [engineering.md](engineering.md) — 63 skills, full delta + AR table + 10 exemplars.
- [engineering-team.md](engineering-team.md) — 52 skills, delta + AR table + the two-
generation role-skill split + 1 new P1 (senior-data-engineer CLI mismatch).
- [improvement-fields.md](improvement-fields.md) — the eight fields where investment moves
the most skills, ordered by leverage (the "what to improve, per field" rollup).
- [research-digest.md](research-digest.md) — the web-verified harness canon.
- [RUBRIC.md](RUBRIC.md) — the AR scoring rubric.
---
## 6. Recommended follow-up PRs (in leverage order)
1. **Loop-cap sweep** — one-sentence stop-condition + iteration cap across the ~15
LOOP-CAPABLE skills at 7–8 points. Cheapest path to ~doubling HARNESS-READY. Use each
domain report's "Top improvement" column as the work list.
2. **Intake sweep** — port the decision-engine "refuse on missing required input" pattern to
the tool-rich TOOL-ONLY skills (AR1 0→2).
3. **Bind the gates** — make described validators *required* (exit-code gate before
proceeding) in llm-wiki, karpathy-coder, docker-development, helm-chart-builder.
4. **New defects** — wire ship-gate's orphaned scanner + fix its 84→89 table; fix
senior-data-engineer's documented CLI; strip senior-ml-engineer's 2024 pricing; fix
claude-coach's 3 June defects; document workflow-builder's by-design `--sample` exit 1.
5. **Deferred structural verdicts** — dedupe the 4 dual-published pairs; merge/retire the
database trio; register named-persona-adversarial-review in the indexes.
6. **CI gate** — add a manifest-drift check (`harness_manifest_builder.py --all
--no-timestamp` + `git diff --exit-code`) so the harness manifests stay true to the tree,
mirroring `derive_counters.py --check`.
Every item above has an executable acceptance criterion in the per-domain reports — the same
"definition of done" discipline the June audit established.
audit/engineering-agentic-2026-07/RUBRIC.md
+41
View File
@@ -0,0 +1,41 @@
# Agentic-Readiness Rubric (AR v1)
Audit date: 2026-07-03 · Branch: `claude/engineering-audit-agentic-loops-hv9x9m`
The June 2026 audit ([../newgen-2026-06/RUBRIC.md](../newgen-2026-06/RUBRIC.md)) asked
"does this skill earn its context window?" This follow-up asks the next question:
**can an agent pick this skill up with a goal and drive it to a verified close?** — the
gather-context → take-action → verify-work → repeat loop the 2024–2026 harness canon
converged on (see [research-digest.md](research-digest.md)).
## The six dimensions (0–2 each, total 0–12)
| # | Dimension | 0 | 1 | 2 |
|---|---|---|---|---|
| AR1 | **Goal intake** | Accepts any input silently | Asks context questions | Forcing questions / intake tool / refuses vague input (exit-code gate) |
| AR2 | **Task decomposition** | No plan step | Prose phases | Explicit planning step or tool whose output the workflow consumes |
| AR3 | **Deterministic execution** | No wired tools | Tools named, CLIs incomplete | Exact runnable CLIs; output consumed by a named next step |
| AR4 | **Verification** | None | Checklist prose | Machine-checkable gate (exit codes, JSON assertions) the workflow REQUIRES before proceeding |
| AR5 | **Loop discipline** | No retry/stop rules | "Re-run until clean" without a cap | Iteration caps, stop conditions, escalation thresholds |
| AR6 | **Close-out** | Work just ends | Informal done statement | Definition of done + state persistence or handoff artifact |
## Classes
| Class | Criteria | Meaning |
|---|---|---|
| **HARNESS-READY** | total ≥ 9 AND AR4 ≥ 1 AND AR5 ≥ 1 | An agent can run this skill inside a bounded loop today |
| **LOOP-CAPABLE** | total 6–8 (or ≥9 failing an AR4/AR5 gate) | One or two targeted additions from harness-ready |
| **TOOL-ONLY** | total 3–5 | Good tools, no loop spine |
| **PROSE-ONLY** | total 0–2 | Knowledge dump; needs structural rebuild |
The AR4/AR5 gate is deliberate: a skill with perfect intake and tools but no verification
gate or stop condition is *more* dangerous in an autonomous loop, not less — it runs
confidently and forever.
## Executable enforcement
The rubric is now mechanized: `engineering/agent-harness/skills/agent-harness/scripts/harness_manifest_builder.py`
records per-skill `agentic_signals` (static evidence for AR1/AR4/AR5/AR6) in every domain
manifest, and `loop_controller.py` enforces AR4/AR5/AR6 at run time regardless of the
skill's own discipline. Improvement PRs should move skills up this ladder; the manifests
make regressions diffable.
audit/engineering-agentic-2026-07/engineering-team.md
+130
View File
@@ -0,0 +1,130 @@
# Domain re-audit: engineering-team/ — delta vs June 2026 + agentic readiness
Audited: 2026-07-03 · 33 `skills/` + 5 standalone packages (52 distinct skills incl.
playwright-pro and self-improving-agent sub-skills). Rubric: [RUBRIC.md](RUBRIC.md).
June baseline: [../newgen-2026-06/engineering-team.md](../newgen-2026-06/engineering-team.md).
## Summary stats
**Delta resolution (16 non-KEEP June verdicts):** RESOLVED 7 · PARTIALLY-RESOLVED 4 ·
STILL-OPEN 5.
- All 4 P0 corrupted-literal sites fixed: grep for `zstringmin1max100` /
`cdnexamplecom` / `click-mei-tobeinthedocument` → **0 hits**.
- google-workspace-cli P0 fixed: `@anthropic/gws` gone; recoordinated to
`npm install -g @googleworkspace/cli`, every reference carries a "verify against your
installed version / pre-v1.0" disclaimer, and `gws_doctor.py` runs gracefully in demo
mode (exit 0).
- 18 stale `.zip` archives at domain root: **deleted** (0 remain).
- **1 NEW P1 defect found** (senior-data-engineer, below). **1 NEW skill found**
(named-persona-adversarial-review — in no index or manifest).
**Agentic-readiness distribution (52 skills):** HARNESS-READY **4** · LOOP-CAPABLE **16** ·
TOOL-ONLY **27** · PROSE-ONLY **5**.
## Scorecard
AR1 intake / AR2 decomposition / AR3 deterministic exec / AR4 verification gate / AR5 loop
discipline / AR6 close-out (0–2 each). Delta status only for skills with non-KEEP June
verdicts. Entries marked \* score ≥9 but fail the HARNESS-READY gate because AR5 = 0.
| Skill | June | Delta | AR1-6 | Tot | Class | Top improvement |
|---|---|---|---|---|---|---|
| senior-fullstack | KEEP | — | 2/2/2/2/1/1 | 10 | **HARNESS-READY** | Add re-run rule after kill-criterion fix ("re-run engine, assert `kill_criteria_tripped` empty") to lift AR5 to 2 |
| tdd-guide | KEEP | — | 1/1/2/2/2/1 | 9 | **HARNESS-READY** | "Bounded Autonomy Rules" + threshold exits are the loop template; add state persistence (write cycle log) for AR6=2 |
| senior-security | CUT-OR-MERGE | **RESOLVED** (rewritten as 64-line STRIDE + router; re-run-is-the-done-signal gate) | 1/1/2/2/1/2 | 9 | **HARNESS-READY** | Add DREAD≥7-without-owner as a machine check (`jq` assertion on threats.json) |
| playwright-pro/fix | KEEP | — | 0/1/2/2/2/2 | 9 | **HARNESS-READY** | Best loop in the domain ("all 10 must pass, else back to step 3"); add an iteration cap (max 3 fix rounds → escalate) |
| red-team | KEEP | — | 2/2/2/2/0/2 | 10\* | LOOP-CAPABLE\* | Add explicit retry/stop rules per engagement phase (e.g., abort criteria on detection) |
| ai-security | KEEP | — | 2/1/2/2/0/1 | 8 | LOOP-CAPABLE | Add remediate→re-scan loop: "re-run scanner after fixes, exit 0 required" |
| security-pen-testing | KEEP | — | 1/2/2/2/0/2 | 9\* | LOOP-CAPABLE\* | AR5=0 blocks HARNESS; add rescan-until-clean loop with cap |
| senior-secops | KEEP | — | 1/2/2/2/1/1 | 9\* | LOOP-CAPABLE\* | Add CVE-SLA-driven stop condition to formalize AR5 |
| threat-detection | KEEP | — | 1/2/2/2/0/1 | 8 | LOOP-CAPABLE | Add tuning loop (adjust baseline, re-run until FP rate < target) |
| cloud-security | KEEP | — | 1/2/2/2/0/1 | 8 | LOOP-CAPABLE | Add fix→re-check-until-exit-0 loop; add close-out DoD |
| incident-response | KEEP | — | 1/2/2/2/0/1 | 8 | LOOP-CAPABLE | Add containment-verification re-run gate |
| code-reviewer | KEEP | — | 1/1/2/2/0/1 | 7 | LOOP-CAPABLE | Regression-fixture pattern is exemplary; add re-review-after-fix loop |
| senior-frontend | OPTIMIZE | **RESOLVED** (corrupted config fixed; decision engine + forcing questions present) | 2/1/2/2/1/1 | 9 | **HARNESS-READY** | Still 572 lines — move React/Next patterns to references/ per June Verify |
| senior-backend | OPTIMIZE | **RESOLVED** (Zod literal fixed; decision engine refuses on missing inputs) | 2/2/2/1/0/1 | 8 | LOOP-CAPABLE | Add executable SLO-floor verification gate + re-run loop (AR4→2, AR5) |
| senior-qa | OPTIMIZE | **PARTIALLY** (corrupted snippets fixed; still `msw rest.` v1 API L274 + `upload-artifact@v3` L220) | 1/1/2/2/0/1 | 7 | LOOP-CAPABLE | Bump msw to `http`/`HttpResponse`, artifact@v4; add TS-block parse gate |
| senior-architect | OPTIMIZE | **PARTIALLY** (tools well-wired; workflow still ends at "document decision") | 1/1/2/1/0/1 | 6 | LOOP-CAPABLE | Add "re-run dependency_analyzer, assert circular=0" close-out gate (AR4/AR5) |
| senior-ml-engineer | OPTIMIZE | **STILL-OPEN** (12 stale-model hits: GPT-4/3.5/Claude-3 2024 pricing in SKILL.md L154-157 + llm_integration_guide.md L181-251) | 1/1/1/2/2/0 | 7 | LOOP-CAPABLE | Strip 2024 pricing/context tables → model-agnostic; wire tool output→next-step |
| senior-prompt-engineer | REWRITE | **RESOLVED** (0 stale-model hits; workflows end in executable eval gates with ≥baseline loop) | 1/1/2/2/2/0 | 8 | LOOP-CAPABLE | Add AR6 close-out (persist eval baseline as DoD artifact); intake forcing questions |
| senior-data-scientist | OPTIMIZE | **RESOLVED** (phantom scripts replaced by real ones) | 1/1/1/1/0/2 | 6 | LOOP-CAPABLE | Add exact CLI invocations + machine-checkable eval gate |
| senior-data-engineer | OPTIMIZE | **STILL-OPEN + NEW P1** (see below) | 0/1/1/0/0/0 | 2 | **PROSE-ONLY** | Fix documented CLI to match actual argparse; add a verification gate |
| stripe-integration-expert | OPTIMIZE | **STILL-OPEN** (pinned `apiVersion: "2024-04-10"` L72; 0 scripts/refs; 476-line code dump) | 0/0/0/0/1/0 | 1 | **PROSE-ONLY** | Replace pinned version with placeholder+instruction; add `stripe trigger` smoke gate |
| email-template-builder | OPTIMIZE | **STILL-OPEN** (439-line single-file dump; no executable gate) | 0/0/0/0/0/2 | 2 | **PROSE-ONLY** | Invert code:rules ratio, move code to references/, add `react-email` render gate |
| incident-commander | OPTIMIZE | **RESOLVED** (3 scripts all wired; security-triage disambiguation added) | 1/1/2/2/0/1 | 7 | LOOP-CAPABLE | Add re-run gate + iteration cap on timeline reconstruction |
| ms365-tenant-manager | OPTIMIZE | **RESOLVED** (all 3 scripts referenced with exact paths) | 1/1/2/1/0/2 | 7 | LOOP-CAPABLE | Add machine-checkable verify gate (CA report-only assertion) |
| tech-stack-evaluator | OPTIMIZE | **STILL-OPEN** (no `data_as_of` field; embedded ecosystem data undated) | 0/0/2/0/0/0 | 2 | **PROSE-ONLY** | Add `data_as_of` to JSON output; wire all 7 scripts; add TCO regression gate |
| engineering-skills (index) | OPTIMIZE | **PARTIALLY** (32 vs actual 33 after new skill; named-persona absent from every index) | 0/0/0/0/0/0 | 0 | **PROSE-ONLY** (index by design) | Retrue to 33; add named-persona row; single source of truth |
| adversarial-reviewer | KEEP | — | 1/2/0/0/1/1 | 5 | TOOL-ONLY | Add a scored verdict tool + machine gate |
| named-persona-adversarial-review | **NEW (not in June)** | n/a | 1/2/0/1/1/1 | 6 | LOOP-CAPABLE | No scripts; has BLOCKER promotion + re-review exit condition. Add a verdict-emitting tool; register in indexes |
| aws-solution-architect | KEEP | — | 1/2/2/1/1/1 | 8 | LOOP-CAPABLE | Add `cfn-lint`/validate-template close-out gate |
| azure-cloud-architect | KEEP | — | 1/2/2/0/0/1 | 6 | LOOP-CAPABLE | Add `az bicep build` verification gate |
| gcp-cloud-architect | KEEP | — | 1/2/2/0/0/2 | 7 | LOOP-CAPABLE | Add IaC validate gate + retry loop |
| epic-design | KEEP | — | 1/2/0/0/0/2 | 5 | TOOL-ONLY | Wire inspect-assets.py output into a pass/fail gate |
| senior-computer-vision | KEEP | — | 1/2/2/0/0/0 | 5 | TOOL-ONLY | Add eval-metric gate (mAP threshold) + close-out |
| senior-devops | KEEP | — | 1/1/2/1/0/0 | 5 | TOOL-ONLY | Add `terraform validate` gate + healthz retry loop as explicit AR5 |
| snowflake-development | KEEP | — | 1/2/1/1/0/0 | 5 | TOOL-ONLY | Add SQL-lint/dry-run gate |
| a11y-audit | KEEP | — | 1/1/1/2/1/1 | 7 | LOOP-CAPABLE | Baseline-compare loop present; add iteration cap |
| google-workspace-cli | REWRITE | **PARTIALLY** (coordinates fixed + disclaimers; tool provenance still unverifiable) | 1/2/2/0/0/0 | 5 | TOOL-ONLY | Add `gws --version` precondition gate; recipe-vs-`--help` validation step |
| playwright-pro/pw | KEEP | — | 0/1/1/2/1/0 | 5 | TOOL-ONLY | Router; fine as-is |
| playwright-pro/init | KEEP | — | 0/1/1/1/1/0 | 4 | TOOL-ONLY | Add config-assertion gate (retries=2 in CI) |
| playwright-pro/generate | KEEP | — | 0/1/1/2/1/0 | 5 | TOOL-ONLY | Strong (reporter=list gate before done); add iteration cap |
| playwright-pro/review | KEEP | — | 0/1/0/2/0/0 | 3 | TOOL-ONLY | Add fix-loop handoff to /pw:fix |
| playwright-pro/migrate | KEEP | — | 0/1/1/0/0/0 | 2 | PROSE-ONLY | Add parity-check gate before decommission (June Verify) |
| playwright-pro/coverage | KEEP | — | 0/2/0/0/0/0 | 2 | PROSE-ONLY | Wire a coverage-report tool + priority-ranked gate |
| playwright-pro/report | KEEP | — | 0/1/1/0/1/1 | 4 | TOOL-ONLY | Add absent-input error gate |
| playwright-pro/testrail | KEEP | — | 0/0/1/0/0/0 | 1 | PROSE-ONLY | Env-var precondition refusal is the AR1 win; document it as a gate |
| playwright-pro/browserstack | KEEP | — | 0/0/1/0/0/0 | 1 | PROSE-ONLY | Same as testrail |
| self-improving-agent (root) | KEEP | — | 0/0/0/0/0/0 | 0 | PROSE-ONLY (overview by design) | No executable surface |
| si/review | KEEP | — | 0/2/0/0/0/0 | 2 | PROSE-ONLY | Add bucket-count assertion gate |
| si/promote | KEEP | — | 1/2/0/0/0/1 | 4 | TOOL-ONLY | Both-halves check (write + remove source) should be a machine gate |
| si/extract | KEEP | — | 0/2/0/0/0/0 | 2 | PROSE-ONLY | Wire `audit_skills.py` no-FAIL gate (June Verify) explicitly |
| si/remember | KEEP | — | 0/2/0/0/0/1 | 3 | TOOL-ONLY | Add timestamp+category write assertion |
| si/status | KEEP | — | 0/2/0/0/0/1 | 3 | TOOL-ONLY | Add 200-line-budget overflow gate |
## Systemic findings
1. **Loop discipline (AR5) is the domain-wide bottleneck.** The security suite (red-team
10, security-pen-testing 9, senior-secops 9) and cloud architects have excellent intake,
deterministic tools, and exit-code gates but almost no explicit retry/stop-condition/
iteration-cap language. Adding a single "remediate → re-run tool → exit 0 required, max
N rounds then escalate" block would promote ~6 skills to HARNESS-READY at low cost.
2. **Best harness exemplars (template these):** playwright-pro/fix ("run `--repeat-each=10`,
all 10 must pass, else back to step 3" — the cleanest verify+loop in the domain);
senior-fullstack (decision engine that *refuses* on missing inputs + forcing-question
library + kill criteria — the new-gen role template); code-reviewer (committed regression
fixtures with expected `--json` output); senior-security rewritten ("the re-run is the
done signal, not the document" — model close-out phrasing); senior-prompt-engineer
(every workflow now ends in an executable gate — a genuine REWRITE→RESOLVED turnaround).
3. **Role skills split into two generations.** UPGRADED (decision engine + forcing questions
+ kill criteria): senior-fullstack, senior-frontend, senior-backend. UN-UPGRADED (no
forcing questions, no decision engine, workflows end at "document"): senior-architect,
senior-devops, senior-qa, senior-data-engineer, senior-data-scientist,
senior-ml-engineer, senior-computer-vision, senior-secops, senior-prompt-engineer. The
three upgraded roles should export a shared loop template — `Assumptions → decision
engine (refuse on missing input) → forcing questions with kill criteria → execute →
re-run engine/tool → assert gate → DoD` — and the nine un-upgraded roles should adopt
it. senior-architect and senior-devops are the highest-value targets.
4. **NEW P1 — senior-data-engineer documents a CLI the tool doesn't have.** SKILL.md L74
shows `data_quality_validator.py validate --checks freshness,completeness,uniqueness
--input …`, but the shipped `validate` subcommand has **no `--checks` and no `--input`**
(input is positional). SKILL.md also invokes `etl_performance_optimizer.py`, which is
**not present** in `scripts/`. Any agent following the docs emits failing commands.
5. **STILL-OPEN staleness — senior-ml-engineer.** 12 hits of GPT-4 / GPT-3.5 / Claude 3
Opus 2024 pricing and "GPT-4 8,192 context" presented as current. The parallel A6 fix
landed for senior-prompt-engineer (0 hits) but not here.
6. **Count drift persists + new skill unregistered.** Actual `skills/` dir = 33 (June said
32). named-persona-adversarial-review (PR #867) appears in **no** index — not
engineering-skills SKILL.md, README, START_HERE, or plugin.json `skills`.
engineering-team/CLAUDE.md still lists **8 phantom script filenames**
(`fullstack_scaffolder.py`, `statistical_analyzer.py`, `etl_generator.py`,
`mlops_setup_tool.py`, `llm_integration_builder.py`, `rag_system_builder.py`,
`video_processor.py`) — June finding #6 unresolved.
7. **Three PROSE-ONLY skills need structural rebuild, not tuning:** stripe-integration-expert
(1/12), email-template-builder (2/12), tech-stack-evaluator (2/12). All were OPTIMIZE in
June and are STILL-OPEN — code/prose dumps with no wired-tool→gate→loop spine.
8. **Intake (AR1) is near-universally absent.** Only 5 skills score AR1≥2 (the upgraded
role trio via decision-engine refusal; red-team, ai-security via authorization gates).
40+ skills have zero forcing-question/refusal-on-vague-input intake. The decision-engine
"refuse without required inputs" pattern is the cheapest AR1 fix to propagate.
audit/engineering-agentic-2026-07/engineering.md
+190
View File
@@ -0,0 +1,190 @@
# Domain re-audit: engineering/ — delta vs June 2026 + agentic readiness
Audited: 2026-07-03 · 63 distinct skills under `engineering/` · Method: full SKILL.md
reads, June "Verify" criteria re-run, ~30 script smoke tests (all exit codes checked).
Rubric: [RUBRIC.md](RUBRIC.md). June baseline: [../newgen-2026-06/engineering.md](../newgen-2026-06/engineering.md).
## Summary stats
**Delta resolution (19 non-KEEP June verdicts):**
- **RESOLVED: 9** — agent-designer, dependency-auditor, rag-architect, skill-tester,
tech-debt-tracker (REWRITEs); command-guide (deleted); release-manager (merged into
changelog-generator, hotfix/rollback tables absorbed); engineering-advanced-skills
(counters now 37=37=37, paths fixed); universal-scraping-architect (all 3 scripts wired,
agent/command rebuilt, layout normalized).
- **PARTIALLY-RESOLVED: 6** — migration-architect, observability-designer (CLIs + gates
added but textbook bodies never pruned); database-designer (wired, not merged);
agent-workflow-designer, api-design-reviewer, runbook-generator.
- **STILL-OPEN: 4** — database-schema-designer (no merge, zero scripts, broken seed example
at L154 persists), codebase-onboarding, interview-system-designer, claude-coach (all 3
June defects untouched: dup frontmatter keys `Name:`+`name:` / `1.0.0`+`2.9.0`, README
paste L145–205, unwired classifier).
- Side-asks: 5 orphan plugins now marketplace-registered ✅ · 4 dual-published duplicates
(slo/chaos/k8s/flags) still undeduped (byte-identical, `diff -rq` clean) ❌ · autoresearch
evaluator `--help`-exception sentence never added ❌ · env-secrets-manager dead cross-refs
persist ❌ · focused-fix `superpowers:*` still "REQUIRED SUB-SKILL" ❌.
**Agentic-readiness distribution (63 skills):** HARNESS-READY **22** · LOOP-CAPABLE **23**
· TOOL-ONLY **16** · PROSE-ONLY **2**. Weakest dimensions: **AR5 loop discipline**
(caps/stop conditions rare outside v2.4+ skills) and **AR1 goal intake** (most tool-rich
skills accept any input silently).
**KEEP spot-checks (~25 contracts re-run):** PASS except — **ship-gate** (category table 84
vs checks.md 89), **write-a-skill** (fails its own checklist runner: 141 lines vs its <100
rule, exit 1), **workflow-builder** (`--sample` exits 1 *by design* — June criterion wrong,
needs one doc sentence), **focused-fix** (superpowers refs unresolved),
**env-secrets-manager** (all 5 cross-ref paths fail `ls`). Verified anchors reproduce: slo
error-budget 43.20 min, statistical-analyst +1.2pp, tc-tracker rejects `planned→deployed`
(exit 2), commit_linter/mcp_validator `--strict` exit 1 correctly.
## Per-skill table
Scores AR1·AR2·AR3·AR4·AR5·AR6. Class: HR=HARNESS-READY, LC=LOOP-CAPABLE, TO=TOOL-ONLY,
PO=PROSE-ONLY. Delta "—" = KEEP verdict holding.
| Skill | June | Delta | AR1-6 | Tot | Class | Top improvement |
|---|---|---|---|---|---|---|
| skills/agent-designer | REWRITE | RESOLVED | 1·2·2·2·1·2 | 10 | HR | Cap the step-4 re-evaluate loop (max 3 pilot re-runs, then escalate) |
| skills/agent-workflow-designer | OPTIMIZE | PARTIAL | 0·1·2·1·0·0 | 4 | TO | Add June-mandated "When NOT to use → workflow-builder" block + JSON-validity gate on scaffolder output |
| skills/api-design-reviewer | OPTIMIZE | PARTIAL | 0·1·2·2·1·1 | 7 | LC | Cut L31-349 textbook REST to references/ (body ≤200); cap lint-fix cycles at 3 |
| skills/api-test-suite-builder | KEEP | — | 0·1·2·0·0·0 | 3 | TO | Add gate: generated suite must pass `npx vitest run`/`pytest -x` with 0 collection errors; coverage contract per route |
| skills/browser-automation | KEEP | — | 1·1·1·1·1·0 | 5 | TO | Full runnable CLIs in Quick Start; require `anti_detection_checker.py` exit-0 pre-run; retry cap 3 on 429/403 |
| skills/changelog-generator | KEEP | — (merge done) | 1·1·2·2·1·2 | 9 | HR | State retry cap for lint-fix cycle |
| skills/chaos-engineering | KEEP | dedupe open | 2·2·2·2·2·2 | 12 | HR | Deduplicate bundle/standalone copies |
| skills/ci-cd-pipeline-builder | KEEP | — | 0·1·2·1·0·1 | 5 | TO | Require `yaml.safe_load` exit-0 gate on generated pipeline; intake (platform/targets/branches) |
| skills/codebase-onboarding | OPTIMIZE | STILL-OPEN | 0·1·2·1·0·1 | 5 | TO | Add June-mandated gate: execute every setup command in the doc once, 0 ❌ before done |
| skills/database-designer | CUT-OR-MERGE | PARTIAL | 1·1·2·2·1·1 | 8 | LC | Execute the merge into sql-database-assistant (or add explicit routing); cap analyze-fix at 2 cycles |
| skills/database-schema-designer | CUT-OR-MERGE | STILL-OPEN | 0·1·0·1·0·0 | 2 | PO | Retire per June verdict: migrate RLS block + pitfalls table, delete broken L154 seed example |
| skills/dependency-auditor | REWRITE | RESOLVED | 0·1·2·2·1·1 | 7 | LC | Intake (ecosystem/policy/threshold); cap upgrade-rescan at 2 cycles |
| skills/engineering-advanced-skills | OPTIMIZE | RESOLVED | 0·0·0·0·0·0 | 0 | PO (index by design) | Optionally add "state which skill you loaded and why" routing rule |
| skills/env-secrets-manager | KEEP | cross-refs STILL-OPEN | 0·1·2·1·1·1 | 6 | LC | Fix 5 dead cross-ref paths; make `env_auditor.py` 0-critical a binding close gate |
| skills/feature-flags-architect | KEEP | dedupe open | 1·2·2·2·2·2 | 11 | HR | Deduplicate copies |
| skills/focused-fix | KEEP | PARTIAL | 2·2·1·2·2·2 | 11 | HR | Reword `superpowers:*` as optional externals; drop phantom `scope` skill ref (L308) |
| skills/full-page-screenshot | KEEP | — | 1·1·2·2·1·1 | 8 | LC | Hard gate: `file out.png` = PNG & height>viewport; stop after 2 `--wait` increases |
| skills/git-worktree-manager | KEEP | — | 1·1·2·2·1·1 | 8 | LC | Make Validation Checklist a required exit gate with one recovery pass then escalate |
| skills/interview-system-designer | OPTIMIZE | STILL-OPEN | 0·1·1·1·0·0 | 3 | TO | Wire or delete the 3 orphan root-level scripts; relocate out of engineering per June misfit note |
| skills/kubernetes-operator | KEEP | dedupe open | 1·1·2·2·1·2 | 9 | HR | Cap validator fix-rerun cycles at 3; deduplicate copies |
| skills/mcp-server-builder | KEEP | — | 1·1·2·2·0·1 | 7 | LC | Loop rule: fix + re-run `mcp_validator.py --strict` until exit 0, max 3 cycles; done contract (paths + JSON keys) |
| skills/migration-architect | REWRITE | PARTIAL | 1·2·2·2·1·1 | 9 | HR | Cut L55-429 textbook to references/ (Verify cap ≤200); stop condition: 3 failed gate revisions → escalate |
| skills/monorepo-navigator | KEEP | — | 1·0·2·0·0·0 | 3 | TO | Numbered workflow + gate (analyzer JSON `cycles` empty; affected-only CI filter); artifact = workspace map |
| skills/observability-designer | REWRITE | PARTIAL | 1·1·2·1·1·1 | 7 | LC | Prune L35-273 golden-signals brochure; make alert loop a hard gate (duplicate count = 0) with 1-rotation stop |
| skills/performance-profiler | KEEP | — | 1·1·2·1·0·1 | 6 | LC | Before/after numbers as required artifact (<10% improvement → revert); one-bottleneck-at-a-time stop rule |
| skills/pr-review-expert | KEEP | — | 1·1·2·1·0·1 | 6 | LC | Verdict gate (BLOCK on MUST-FIX or coverage < −5%); re-review loop max 3 rounds then human |
| skills/rag-architect | REWRITE | RESOLVED | 1·2·2·2·2·2 | 11 | HR | Move 3 root scripts into scripts/ (layout anomaly only) |
| skills/runbook-generator | OPTIMIZE | PARTIAL | 1·1·2·1·0·1 | 6 | LC | Add June-required post-generation checklist as refusal gate (rollback non-empty, every step has verify line) |
| skills/secrets-vault-manager | KEEP | — | 1·1·1·1·0·0 | 4 | TO | Exact CLIs for all 3 scripts; gate: audit_log_analyzer shows zero old-credential usage before rotation done |
| skills/self-eval | KEEP | — | 1·1·0·2·1·2 | 7 | LC | Prompt-only by design; optional tiny `scores_check.py` JSONL assertion to formalize AR3 |
| skills/ship-gate | KEEP | table-drift FAIL | 2·2·0·2·2·2 | 10 | HR | Wire the fully orphaned `ship_gate_scanner.py` (~1230 LOC) as Step 2 with exit-code verdict; true-up table 84→89 |
| skills/skill-security-auditor | KEEP | — | 1·1·2·2·0·1 | 7 | LC | Remediate→re-scan loop until PASS (max 3); attach JSON report to install decision |
| skills/skill-tester | REWRITE | RESOLVED | 1·1·2·2·2·1 | 9 | HR | Recalibrate `skill_validator.py` tier minimums (still scores new-style <100-line skills POOR) |
| skills/slo-architect | KEEP | dedupe open | 2·1·2·2·1·2 | 10 | HR | Deduplicate copies (bundle Quick Start points at standalone path — deleting standalone breaks bundle docs) |
| skills/spec-driven-workflow | KEEP | — | 2·2·1·2·2·2 | 11 | HR | Fix pathless CLIs (`python spec_validator.py` → `python3 scripts/spec_validator.py`, L151/177/324-333) → AR3=2 |
| skills/sql-database-assistant | KEEP | — | 0·1·2·1·0·0 | 4 | TO | Gate every generated query through `query_optimizer.py` (score <70 → rewrite); fix phantom `observability-platform` ref |
| skills/tc-tracker | KEEP | — | 1·1·2·2·1·2 | 9 | HR | Already strong; add explicit iteration cap on validation-fix loop |
| skills/tech-debt-tracker | REWRITE | RESOLVED | 1·1·2·2·1·1 | 8 | LC | Stop condition (2 flat snapshots → re-prioritize) + sprint artifact contract → ≥9 |
| agenthub (8 sub-skills) | KEEP | — | 2·2·2·2·2·2 | 12 | HR | Wire orphaned `dry_run.py` as mandatory pre-spawn gate in /hub:run; explicit max-attempt cap |
| autoresearch-agent (6) | KEEP | doc-ask STILL-OPEN | 2·2·2·2·2·2 | 12 | HR | Add evaluator `--help`-exception sentence (June ask); replace stale CronCreate/CronDelete tool names |
| behuman | KEEP | registered ✅ | 1·1·0·1·1·1 | 5 | TO | Ship a mirror-check lint as pre-output gate; cap the rewrite loop |
| caveman | KEEP | — | 0·0·1·1·1·1 | 4 | TO | Inline the 3 exact `caveman_lint.py` invocations (now only in companion_tooling.md); require PASS/WARN before sending |
| claude-coach | OPTIMIZE | STILL-OPEN (all 3) | 2·1·1·1·2·1 | 8 | LC | Fix dup frontmatter + delete L145-205 README paste; wire `coach_tip_classifier.py` as Rule-5 gate → HR |
| code-tour | KEEP | — | 1·1·0·1·0·2 | 5 | TO | Add `tour_validator.py` (schema + file/line existence, exit 0 before save); max 2 re-verify passes |
| collab-proof | NEW | — | 2·2·2·2·1·2 | 11 | HR | Add retry/cap rule for token-collection fallback; translate leftover Korean rubric phrases |
| data-quality-auditor | KEEP | — | 1·1·2·2·0·2 | 8 | LC | Remediate→re-profile loop with DQS delta report, cap 3 cycles → HR |
| demo-video | KEEP | — | 1·1·0·1·0·2 | 5 | TO | Ship scenes.json validator required before build.sh; exact ffmpeg fallback commands |
| docker-development | KEEP | — | 0·1·2·2·0·1 | 6 | LC | Intake (Dockerfile path + size/speed/security target); analyzer-score-must-improve loop, max 3 passes |
| grill-me | KEEP | — | 2·2·1·1·2·2 | 10 | HR | Exact flags for extractor/generator CLIs; machine gate = session JSON all branches `resolved` |
| grill-with-docs | KEEP | registered ✅ | 2·2·2·2·2·2 | 12 | HR | None blocking — exemplar |
| handoff (engineering) | KEEP | — | 1·1·1·0·0·2 | 5 | TO | Wire 3 scripts with exact CLIs; port sibling productivity/handoff `handoff_self_check.py` 6-check gate |
| helm-chart-builder | KEEP | — | 1·1·2·2·0·1 | 7 | LC | Fix-and-revalidate loop (`chart_analyzer.py` 0 CRITICAL, cap 3); intake (workload/namespace/secrets) |
| karpathy-coder | KEEP | — | 1·2·1·1·1·0 | 6 | LC | Exact CLIs in SKILL.md (only agent/command carry them); make pre-commit check a required gate not warn-only |
| llm-cost-optimizer | KEEP | registered ✅ | 2·1·0·2·1·1 | 7 | LC | Add one stdlib script (savings estimator) with CLI; before/after cost-JSON gate between techniques |
| llm-wiki | KEEP | — | 1·1·2·1·0·2 | 7 | LC | Make `lint_wiki.py` exit code a required post-ingest gate (now "periodic"); fix 4 phantom related-skill refs |
| prompt-governance | KEEP | registered ✅ | 2·1·0·1·1·1 | 6 | LC | Ship registry-YAML validator; golden-dataset minimum (20) as Mode-2 refusal gate |
| security-guidance | KEEP | — | 0·0·1·2·0·1 | 5 | TO (hook by design) | Optional `--scan <file>` manual mode for deterministic re-run-to-exit-0 |
| statistical-analyst | KEEP | — | 2·1·2·2·1·1 | 9 | HR | Add H1 heading (currently none); cap extend/re-test loop at one extension |
| terraform-patterns | KEEP | — | 0·1·2·1·0·1 | 5 | TO | Gate: 0 Critical from `tf_security_scanner.py --strict` before apply; fix `./scripts/convert.sh` path |
| universal-scraping-architect | OPTIMIZE | RESOLVED | 1·1·2·2·1·1 | 8 | LC | Promote agent's intake to SKILL.md forcing questions; cap re-extraction at 2 attempts → HR |
| workflow-builder | KEEP | — (contract nuance) | 2·2·2·2·2·1 | 11 | HR | Document that `validate_workflow.py --sample` exits 1 by design; add done digest |
| write-a-skill | KEEP | self-check FAIL | 2·1·2·2·1·2 | 10 | HR | Trim own SKILL.md to <100 lines so it passes its own checklist runner; make runner exit-0 a blocking Phase-3 gate |
| zero-hallucination-coder | NEW | — | 2·2·0·2·2·2 | 10 | HR | Add one stdlib plan-linter (scan for unresolved `[UNKNOWN]`/TODO, exit 1) — AR3=0 is the only gap |
(`named-persona-adversarial-review` is scored in [engineering-team.md](engineering-team.md) —
it lives at `engineering-team/skills/`.)
## Systemic findings
### Patterns
1. **The REWRITE wave worked, but two were patches, not rewrites.** Brochure headings
("Future Enhancements"/"Conclusion"/"Planned Features") are now zero across engineering/;
5 of 7 REWRITEs fully resolved. migration-architect (429 lines) and observability-designer
(272) got a wired Quick Start + gate bolted onto an unpruned textbook body — the June
Verify line caps remain unmet.
2. **AR5 (loop discipline) is the domain's weakest muscle.** Only v2.4+/Pocock/orchestrator-
generation skills state iteration caps or stop conditions. ~20 skills have a "re-run until
clean" instruction with no cap; ~25 have none at all. A single sentence pattern ("max N
fix-rerun cycles, then escalate") would lift 8 skills sitting at 7–8 into HARNESS-READY.
3. **AR1 (goal intake) missing from tool-rich skills.** The wiring epidemic was fixed (AR3
median is now 2), but most wired skills run on whatever input arrives — no forcing
questions, no refusal on vague goals. The intake patterns already exist in-house
(workflow-builder, grill-me, zero-hallucination-coder) and just need porting.
4. **Gates exist but aren't binding.** Many skills *describe* a validator yet don't
*require* its exit code before proceeding (llm-wiki "periodic" lint, karpathy-coder
warn-only hook, docker/helm validate-steps without loop closure).
5. **Orphan scripts persist even in KEEP skills** — new cases surfaced: **ship-gate's
`ship_gate_scanner.py` (~1230 LOC, full exit-code contract, never mentioned in SKILL.md —
the model is told to scan manually)**, agenthub's `dry_run.py`, interview-system-designer's
3 root scripts, secrets-vault-manager's 3 unwired tools, handoff's 3 tools.
6. **Dead cross-references survived the phantom-path sweep** because they're skill-name
table refs, not file paths: env-secrets-manager (5 dead), sql-database-assistant
(`observability-platform`), llm-wiki (4 phantom related skills), focused-fix (`scope` +
`superpowers:*` as REQUIRED).
7. **Dual-published dedupe (June finding #4) not executed.** All 4 pairs remain
byte-identical (no divergence yet); trap: bundle copies' Quick Starts reference the
*standalone* paths, so naive deletion of standalone copies breaks the bundle's own docs.
8. **Database-trio merge (June finding #3a) not executed** — database-designer got wired
instead of merged; database-schema-designer remains the domain's worst skill (PROSE-ONLY,
broken seed example intact).
### New defects found this audit
- ship-gate: orphaned scanner + category-table drift (SKILL.md 84 vs checks.md 89) — its
KEEP contract now FAILS.
- write-a-skill: fails its own checklist runner (141 lines vs its <100 rule; exit 1) —
the meta-skill doesn't dogfood.
- autoresearch loop sub-skill: instructs stale `CronCreate`/`CronDelete` tool names and a
10-min interval that conflicts with the current hourly-minimum trigger surface — broken
as written.
- claude-coach: zero progress on all 3 June defects (conflicting `Version: 1.0.0` /
`version: 2.9.0` still parses ambiguously).
- workflow-builder: `validate_workflow.py --sample` exits 1 *by design* (intentionally
broken sample) — the June KEEP criterion assumed 0; needs one documenting sentence.
- database-schema-designer: broken seed example at L154 persists (explicit June Verify item).
- collab-proof: untranslated Korean rubric phrases from the upstream Vela source.
- terraform-patterns: `./scripts/convert.sh` invocation resolves only from repo root; repo
version `2.9.0` leaked into an Infracost policy example.
- statistical-analyst: no H1 heading; "You are an expert…" opener also in
data-quality-auditor.
- skill-tester's `skill_validator.py` still penalizes new-style <100-line skills (scores
self-eval "POOR 33.3") despite the doc-level scope note.
- No broken scripts: all ~60 `--help`/`--sample`/pipe invocations exited per contract
(non-zero only where documented).
### Top-10 harness-ready exemplars
**agenthub (12)**, **autoresearch-agent (12)**, **chaos-engineering (12)**,
**grill-with-docs (12)**, **collab-proof (11)**, **feature-flags-architect (11)**,
**spec-driven-workflow (11)**, **workflow-builder (11)**, **rag-architect (11)**,
**focused-fix (11)**. Honorable mentions at 10: slo-architect, ship-gate, grill-me,
zero-hallucination-coder, write-a-skill — each one small fix from exemplar status.
### Highest-leverage next PRs
1. Wire ship-gate's scanner + fix its category table.
2. One-sentence loop-cap sweep across the eight 7–8-point LOOP-CAPABLE skills
(claude-coach, data-quality-auditor, universal-scraping-architect, docker-development,
helm-chart-builder, mcp-server-builder, tech-debt-tracker, skill-security-auditor) —
the cheapest path to ~30 HARNESS-READY.
3. Execute the deferred structural verdicts: dedupe the 4 dual-published pairs, merge/retire
the database trio, fix claude-coach.
audit/engineering-agentic-2026-07/improvement-fields.md
+88
View File
@@ -0,0 +1,88 @@
# Improvement fields — what to fix, per field, across both engineering domains
The per-skill line items live in [engineering.md](engineering.md) and
[engineering-team.md](engineering-team.md). This file rolls them up into the **eight
fields** where investment moves the most skills, ordered by leverage (skills lifted per
unit of work). Distribution today, 115 distinct skills across both domains:
**26 HARNESS-READY · 39 LOOP-CAPABLE · 43 TOOL-ONLY · 7 PROSE-ONLY.**
## Field 1 — Loop discipline (AR5): the single biggest lever
~45 skills either say "re-run until clean" with no cap or have no retry/stop language at
all. The fix is one standardized block per skill:
> Remediate → re-run `<tool>` → exit 0 required. Max **3** cycles; a repeat failure for the
> same reason after 2 attempts means a structural assumption is wrong — stop and escalate
> with the evidence log.
- Cheapest wins (already 7–8 points, one sentence from HARNESS-READY):
claude-coach, data-quality-auditor, universal-scraping-architect, docker-development,
helm-chart-builder, mcp-server-builder, tech-debt-tracker, skill-security-auditor
(engineering/); ai-security, threat-detection, cloud-security, incident-response,
senior-secops, red-team, security-pen-testing (engineering-team/).
- Estimated movement: **~15 skills → HARNESS-READY** from this field alone.
## Field 2 — Goal intake (AR1): refuse to run on fuzz
40+ skills accept any input silently. Three proven in-house patterns to propagate:
1. Decision-engine refusal (senior-fullstack/frontend/backend): refuse when required
inputs are missing, list them.
2. Forcing-question library with recommended answers (grill-me, the fork-orchestrators,
zero-hallucination-coder).
3. Exit-code intake gates (agent-harness `goal_compiler.py` exits 3 on vague goals).
Priority targets: every security skill (authorization scope!), ci-cd-pipeline-builder,
docker-development, helm-chart-builder, dependency-auditor, sql-database-assistant.
## Field 3 — Binding verification (AR4): described ≠ required
Validators exist but their exit codes aren't load-bearing. Convert "run the validator"
into "the workflow does not proceed past step N until `<cmd>` exits 0":
- llm-wiki (`lint_wiki.py` is "periodic"), karpathy-coder (warn-only pre-commit),
docker/helm (validate steps without loop closure), senior-architect ("document decision"
instead of "re-run dependency_analyzer, assert circular=0"), cloud architects (no
`cfn-lint` / `az bicep build` / terraform-validate gates), api-test-suite-builder
(generated suite never executed).
## Field 4 — Orphan and mismatched tooling (AR3): the recurring A3 debt
- **New P1:** senior-data-engineer's documented CLI doesn't match the shipped argparse
(`--checks`/`--input` don't exist; `etl_performance_optimizer.py` missing).
- **Worst orphan:** ship-gate's `ship_gate_scanner.py` (~1230 LOC, full exit-code contract,
never mentioned in its SKILL.md).
- Others: agenthub `dry_run.py`, interview-system-designer (3 root scripts),
secrets-vault-manager (3), engineering/handoff (3), spec-driven-workflow (pathless CLIs).
- Prevention: extend CI gate G2 to assert every `scripts/*.py` basename appears in its
SKILL.md (the harness manifests already record `wired: true/false` per tool — a
one-line CI check over the manifests catches this class forever).
## Field 5 — Close-out & state (AR6): make "done" an artifact
Most skills just end. Adopt tc-tracker's handoff block or the agent-harness close contract
(`close` refuses while unverified; emits evidence log + waivers). Targets: performance-
profiler (before/after numbers as required artifact), senior-prompt-engineer (persist eval
baseline), monorepo-navigator (workspace map artifact), all nine un-upgraded senior-* roles.
## Field 6 — Structural verdicts deferred since June
1. Dedupe the 4 dual-published pairs (slo/chaos/k8s/flags) — mind the trap: bundle Quick
Starts reference standalone paths.
2. Merge/retire the database trio (database-schema-designer is still PROSE-ONLY with the
broken L154 seed example).
3. Rebuild the three engineering-team PROSE-ONLY dumps: stripe-integration-expert,
email-template-builder, tech-stack-evaluator.
4. claude-coach's three June defects (dup frontmatter, README paste, unwired classifier).
## Field 7 — Freshness & correctness spot-fixes
- senior-ml-engineer: 12 stale-model hits (2024 GPT-4/Claude-3 pricing as current).
- senior-qa: msw v1 API + `upload-artifact@v3`.
- autoresearch loop sub-skill: stale `CronCreate`/`CronDelete` tool names + interval below
the current hourly trigger minimum.
- stripe-integration-expert: pinned `apiVersion: "2024-04-10"`.
- collab-proof: untranslated Korean rubric phrases.
- Registry drift: named-persona-adversarial-review in zero indexes; engineering-team
CLAUDE.md lists 8 phantom script names; ship-gate table 84 vs 89.
- write-a-skill fails its own checklist runner (dogfooding gap).
## Field 8 — Role-skill loop template (engineering-team's bi
audit/engineering-agentic-2026-07/research-digest.md
+85
View File
@@ -0,0 +1,85 @@
# Research digest: agent harnesses & agentic loops, 2025–2026 best practice
Compiled 2026-07-03 from web-verified sources. This digest informed the AR rubric
([RUBRIC.md](RUBRIC.md)) and the `engineering/agent-harness` skill's design; the full
per-source treatment lives in that skill's `references/` (3 docs, 21 citations).
## 1. The canonical loop
- **Anthropic, "Building Effective Agents" (Schluntz & Zhang, Dec 2024)** — workflows
(predefined code paths) vs agents (model directs its own process); patterns: prompt
chaining with gates, routing, parallelization, orchestrator-workers, evaluator-optimizer
("only when clear evaluation criteria exist"). Start simple; stopping conditions mandatory.
- **Anthropic, Claude Agent SDK (Sep 2025)** — the loop is **gather context → take action →
verify work → repeat**; filesystem as context store; verification ladder: rules-based >
visual > LLM-as-judge.
- **Anthropic, multi-agent research system (Jun 2025)** — subagent specs need objective,
output format, tool guidance, and boundaries; effort scaled by rule (simple = 1 agent,
3–10 calls) because early agents "spawned 50 subagents for simple queries".
- **Anthropic, "Effective harnesses for long-running agents" (Nov 2025)** — initializer
expands the goal into `feature-list.json` (description + acceptance criteria + status);
a worker wakes repeatedly, one feature per fresh-context session; all state on disk/git.
- **Anthropic, Agent Skills (Oct 2025)** — progressive disclosure (metadata → SKILL.md →
files on demand); deterministic scripts for anything reliably automatable; build skills
from observed agent failures.
## 2. Verification discipline
- **Jason Wei, "verifier's law" (Jul 2025)** — training/iterating AI on a task is
proportional to its verifiability; invest in checks before agents.
- **SWE-agent (NeurIPS 2024)** — the highest-value guardrail was a linter rejecting invalid
edits at write time; agents fail when the environment gives no feedback.
- **SWE-bench Verified (OpenAI 2024)** — even benchmark tests were too noisy without human
validation; checks need declared reliability classes.
- **Claude Code best practices (Cherny, Apr 2025)** — strongest loop is test-driven: write
the check first, confirm it fails, iterate against it.
- **Reflexion (Shinn 2023) + Huang et al. (ICLR 2024)** — self-critique helps only when
grounded in external feedback; intrinsic self-correction often degrades answers.
⇒ **Deterministic validators are the primary gate; LLM-as-judge is a fallback.**
- **Anthropic reward-hacking research (Nov 2025)** — agents that game their checks
generalize to worse behavior ⇒ the worker must never adjudicate or modify its own gates.
## 3. Loop patterns in production
- **Ralph Wiggum loop (Huntley, Jul 2025; now an official Claude Code plugin)** — same
prompt to a fresh-context agent in a `while true` loop; filesystem + TODO + git as memory.
Fresh context each iteration is the point; caps and completion criteria are added by
practice.
- **Cognition, "Don't Build Multi-Agents" (Jun 2025)** — conflicting parallel decisions are
the dominant multi-agent failure ⇒ **fan out readers/judges, serialize writers**.
- Caps as runtime errors: OpenAI Agents SDK `max_turns` / guardrail tripwires; LangGraph
`recursion_limit`; Anthropic effort budgets.
## 4. State + memory
- Single JSON state file, atomic writes, schema version; narrative handoff separate from
machine state; git as checkpoint layer; compaction with explicit preserve-lists
(Anthropic context-engineering, Sep 2025; LangGraph checkpointers).
## 5. Failure modes → mitigations
| Failure | Mitigation |
|---|---|
| Infinite loops / runaway effort | Triple cap: iterations, wall-clock, budget — breach = terminal state, never silent |
| Verification theater / reward hacking | Gates read-only to the worker; controller re-runs checks itself; diff-scan for edits to test/gate paths |
| Goal drift / conflicting decisions | Single-writer rule; full-context handoffs |
| Context rot / silent truncation | Fresh-context iterations against durable disk state |
## 6. Manifest designs (goals → skills → verifications)
- **AGENTS.md** (agents.md, Aug 2025; Agentic AI Foundation / Linux Foundation, Dec 2025) —
prose manifest for "how to build and verify here".
- **feature-list.json** (Anthropic long-running harness) — the closest published
goal→tasks→verification manifest.
- **MCP** — declared tool registries as the harness's action space.
- GitHub Agentic Workflows / claude-code-action — declarative agent jobs with permissions +
tool allowlists.
## Consensus (what this repo now implements)
Compile goals into explicit task lists with acceptance criteria; run stateless
fresh-context iterations against durable disk/git state; gate every promotion on
deterministic, agent-untouchable checks; serialize writes, parallelize reads; cap
everything; declare the goal→skill→verification mapping in a per-domain manifest.
Implemented as `engineering/agent-harness` (manifest builder + goal compiler + loop
controller, 18 committed domain manifests).
audit/newgen-2026-06/00-MASTER.md
+117
View File
@@ -0,0 +1,117 @@
# Master Audit Report — New-Generation Model Optimization
**Audited:** 2026-06-10 · **Branch:** `claude/skills-plugins-audit-vrttx1` · **Scope:** every canonical skill, plugin, agent, slash command, script, and registry surface (sync copies under `.codex/ .gemini/ .hermes/ .vibe/` excluded as derived artifacts).
**Method:** two automated sweep layers (the repo's own `audit_skills.py` checklist + a custom new-gen sweep for triggers, verification loops, placeholders, stale models, dead links, duplicate content) followed by 10 parallel domain deep-dives that read every SKILL.md, spot-checked references and scripts, and **executed** empirical claims where the skills make them. Rubric: [RUBRIC.md](RUBRIC.md).
---
## 1. Repo-wide scorecard
**324 unique skills** audited (346 SKILL.md files minus dual-published copies and meta/index files counted once):
| Verdict | Count | % | Meaning |
|---|---|---|---|
| **KEEP** | 190 | 59% | Ships as-is; verification criteria recorded per skill in domain reports |
| **OPTIMIZE** | 102 | 31% | Targeted edits (wiring, triggers, freshness) — content core is sound |
| **REWRITE** | 16 | 5% | Structure salvageable, content is not |
| **CUT-OR-MERGE** | 16 | 5% | Does not earn its context window |
Per domain (links go to the detailed reports with **per-skill custom verification criteria**):
| Domain report | Skills | KEEP | OPT | REW | CUT | Health |
|---|---|---|---|---|---|---|
| [productivity + markdown-html](productivity-markdown-html.md) | 11 | 9 | 2 | 0 | 0 | ★ best — 24/27 live checks pass |
| [research + research-ops](research.md) | 13 | 5 | 7 | 1 | 0 | research-ops all-KEEP; research/ needs wiring |
| [bizops + commercial + finance + growth](bizops-commercial-finance.md) | 24 | 14 | 8 | 0 | 2 | v2.8.0 verified; finance/ legacy |
| [c-level-advisor](c-level-advisor.md) | 61 | 40 | 20 | 0 | 1 | strong content, broken wiring |
| [engineering](engineering.md) | 63 | 44 | 8 | 7 | 4 | two generations coexist |
| [engineering-team](engineering-team.md) | 51 | 35 | 13 | 2 | 1 | code-corruption + stale-era issues |
| [compliance (ra-qm + compliance-os)](compliance.md) | 26 | 13 | 11 | 1 | 1 | P0 regulatory staleness |
| [marketing](marketing.md) | 49 | 20 | 24 | 1 | 4 | orphan scripts + path schism |
| [product-team + project-management](product-pm.md) | 26 | 10 | 9 | 4 | 3 | fabricated MCP wiring |
| [cross-cutting infra](cross-cutting.md) | — | — | — | — | — | registries, CI, root agents/commands |
**Other artifact classes:** 92 agents (17 of 32 root agents lack trigger descriptions; 7 of 13 c-level personas cite phantom reference files; 2 missing frontmatter) · 99 commands (9 root commands are cut/merge candidates; 28 of 39 root commands invoke phantom script paths) · 77 plugin manifests (all schema-valid; **11 not registered in marketplace.json**) · 593 scripts (583 pass `--help`; 1 real crash; 9 by-design).
---
## 2. P0 — Correctness defects (fix before anything else)
These cause a model following the skill to produce **wrong or dangerous output today**:
| # | Defect | Where | Evidence |
|---|---|---|---|
| P0-1 | **Repealed regulation taught as current law.** QSR (21 CFR 820 subsections) presented as in force; QMSR (effective 2026-02-02) never mentioned. Reference + `qsr_compliance_checker.py --section 820.30` built on removed section numbers. | `ra-qm-team/skills/fda-consultant-specialist` | [compliance.md](compliance.md) |
| P0-2 | **ALARP table violates EU MDR.** Risk-acceptability framework includes "cost-benefit of further reduction," which MDR Annex I + EN ISO 14971:2019/A11 prohibit. A notified body would flag this exact table. | `ra-qm-team/skills/risk-management-specialist` | [compliance.md](compliance.md) |
| P0-3 | **EU AI Act Article 5 mis-taught.** Default sample classifies *retail* emotion recognition as prohibited; Art. 5(1)(f) covers workplace/education only. | `ra-qm-team/skills/eu-ai-act-specialist` | [compliance.md](compliance.md) |
| P0-4 | **Silent zero-output finance tools.** All 4 scripts read the wrong JSON shape vs their own bundled sample: zero ratios, $0.00 forecasts, error message then `exit 0`. Invisible to `--help` smoke tests. | `finance/financial-analyst` | [bizops-commercial-finance.md](bizops-commercial-finance.md) |
| P0-5 | **Contradictory discount-margin math.** SKILL.md states the correct fixed-COGS formula; `deal_scorer.py` + reference use a different one (script docstring writes the correct formula, then discards it). Margin dimension (weight 0.30) understates discount damage. | `commercial/deal-desk` | [bizops-commercial-finance.md](bizops-commercial-finance.md) |
| P0-6 | **Corrupted code examples from a past bulk YAML-quoting sweep.** E.g. `z.string().min(1).max(100)` → `"zstringmin1max100"`. 4 confirmed sites; models copying these emit broken code. | `engineering-team` senior-qa:123/244, senior-backend:253, senior-frontend:425 | [engineering-team.md](engineering-team.md) |
| P0-7 | **Fabricated MCP tool names.** 3 Atlassian skills + project-management/CLAUDE.md document four different invented naming conventions; none match the bundled Remote MCP's real tools (`createJiraIssue`, `searchJiraIssuesUsingJql`, …). Every documented tool call fails. | `project-management/` (jira-expert, confluence-expert, atlassian-templates) | [product-pm.md](product-pm.md) |
| P0-8 | **Fabricated install coordinates.** Whole skill teaches a `gws` CLI from `npm i -g @anthropic/gws` / `github.com/googleworkspace/cli` — both almost certainly nonexistent; 43 recipes + 5 wrappers unusable. | `engineering-team/skills/google-workspace-cli` | [engineering-team.md](engineering-team.md) |
| P0-9 | **Skill instructs invoking agents that don't exist in this repo** (documents a different ecosystem: `planner`, `/build-fix`, …). | `engineering/skills/command-guide` | [engineering.md](engineering.md) |
| P0-10 | **Stale orchestrator text bypasses shipped converters.** "v2.10.0 foundation" wording tells the model to hand-render HTML instead of routing to md-document/md-review/md-slides, which shipped. Behavioral, not cosmetic. | `markdown-html/` orchestrator (+ domain CLAUDE.md, README) | [productivity-markdown-html.md](productivity-markdown-html.md) |
| P0-11 | **Script crash:** no argparse; `--help` (or any flag) treated as input filename → FileNotFoundError. Skill's only tool. | `marketing-skill/skills/webinar-marketing/scripts/webinar_funnel_scorer.py` | [marketing.md](marketing.md) |
---
## 3. P1 — Systemic wiring failures (one fix clears many skills)
1. **Phantom-path epidemic (the single biggest repo-wide defect).** A directory reorg added a `skills/` path segment; references never followed. **28 of 39 root commands** invoke scripts at `<domain>/<skill>/scripts/…` that live at `<domain>/skills/<skill>/scripts/…`; same stale shorthand in agents' `skills:` frontmatter, `orchestration/ORCHESTRATION.md` (points at skills that exist only as .zip archives), product/PM routers, c-level persona agents (~16 phantom reference filenames across 7 agents), email agents (`engineering/email/...` → `productivity/email/`), and **5 research skills whose final verification step calls `scripts/office/validate.py` — a file that exists nowhere**. → Build a path-existence linter (see §5) and fix in one sweep.
2. **Orphan-script epidemic.** 40+ working, `--help`-passing scripts are never named in their own SKILL.md (24 in marketing, 8 in product/PM, 6+ in engineering, reflect's all-3, ms365, incident-commander). The model loading these skills cannot use their best assets. → A3 wiring pass: exact CLI + consume-the-output step per script.
3. **Counter drift is structural.** Seven surfaces disagree (README 338 skills / CLAUDE.md header 338 / CLAUDE.md v2.10.3 block 343 / marketplace.json description 64 plugins while containing 66 / actual: **346-347 skills, 66 registered + 11 unregistered plugins, 555 tools, 700 references, 17 domains**). agents/CLAUDE.md says 16 agents; folder has 32. → Derive all counters from the tree via script; never hand-edit again.
4. **11 shippable plugins not in marketplace.json** — compliance-os (advertised in the marketplace's own description yet uninstallable), snowflake-development, behuman, claude-coach, grill-with-docs, llm-cost-optimizer, prompt-governance, business-investment-advisor, video-content-strategist, 2× ra-qm compliance-team.
5. **Marketing context-file path schism.** 19 skills read `.claude/product-marketing-context.md`, 16 read `marketing-context.md`, the creator skill writes `.agents/marketing-context.md`. The domain's "read context first" pattern silently no-ops for half its consumers.
6. **c-level role-registry drift.** Routing tables (agent-protocol, chief-of-staff, board-meeting, founder-mode, brief, c-level-agents frontmatter) stopped at 9 roles; domain has 14. CCO/CDO/CAIO/VPE questions silently misroute. Plus three competing decision-memory architectures and two onboarding interviews writing different schemas to the same file.
7. **Dual-publishing without a guard.** 11 byte-identical skill pairs (engineering ×4, c-level ×5, ra-qm ×2). Zero drift today — but no sync script and no CI check; drift is a matter of time.
8. **Trigger-description gap.** 79 skills and 77 of 92 agents lack "use when" phrasing — weak auto-invocation for new-gen models. Root cause for agents: `templates/agent-template.md` mandates a sub-150-char description with no trigger requirement. 8 v2.8.0 descriptions exceed the 1024-char spec limit (knowledge-ops 1,314 — this, not content, is why it "scored worst").
9. **Meta-tooling violates its own rubric.** `audit_skills.py --help` runs the full 30s audit; `generate-docs.py --help` **rewrites docs/ as a side effect** (verified live — this explains the dirty docs/ files found at session start); hermes/vibe/gemini sync scripts, convert.sh, and generate-docs don't know `markdown-html/` exists; CI's blocking `compileall` skips 8 post-v2.7 domains.
10. **35 stale .zip archives** in the public tree (engineering-team 18, compliance 12, marketing 5) plus 4 internal planning docs in the marketing plugin root.
---
## 4. What's already excellent (the template to copy)
- **research-ops/** — every hard rule verified in execution: ESTIMATE banners, dual-method TAM with triangulation-failure flag, anecdote-vs-insight recurrence gate, named-owner routing on every output. All-KEEP.
- **markdown-html/ + productivity/** — 24/27 empirical checks passed live: exit-code refusal gates, WCAG-AA validation, redaction linter (16 patterns, not the claimed 17 — fix the count), idempotent injection, kill-gate behavior in andreessen.
- **v2.8.0 bizops/commercial orchestrators** — `context: fork` signal-table routing with 2-signal thresholds and no-silent-chain gates verified real, not prose.
- **v2.2 security suite, playwright-pro, self-improving-agent, slo-architect, chaos-engineering, karpathy-coder, Pocock ports** — exit-code contracts, forcing questions, kill criteria, exact CLIs.
- **compliance-os layer** — Article-cited verdicts, explicit NOT-boundaries, outside-counsel routing; **no skill in the repo auto-decides compliance verdicts**.
The repo's quality story is generational, not random: everything built v2.4+ with forcing questions, refusal gates, and wired tools is KEEP-grade; v2.0–v2.1-era skills are capability brochures. The optimization play is to **retrofit the new-gen pattern onto the old generation, not to invent anything new**.
---
## 5. Recommended verification harness (CI gates to add)
Each gate below is the generalized guardrail derived from a defect class this audit found. Suggested home: `scripts/` + `ci-quality-gate.yml`.
| Gate | Catches | Spec |
|---|---|---|
| **G1 path-existence linter** | P1-1 phantom paths | Extract every `scripts/…`, `references/…`, `skills:` and relative-path mention from SKILL.md/agents/commands; assert the file exists. Fails today on ~40 surfaces; burn down, then make blocking. |
| **G2 semantic `--sample` smoke** | P0-4 silent zero-output | For every script with `--sample`/bundled sample data: run it, assert exit 0 **and** output passes a per-skill assertion (non-zero metric count, required JSON keys, banner strings). Per-skill assertions are already written: see "Verify (definition of done)" blocks in every domain report. |
| **G3 counter derivation** | P1-3 drift | `scripts/derive_counters.py` counts skills/plugins/tools/refs/agents/commands from the tree and rewrites the counter blocks in README/CLAUDE.md/marketplace.json. CI fails if claimed ≠ derived. |
| **G4 dual-publish drift guard** | P1-7 | `diff -rq` the 11 known pairs; fail on first divergence. |
| **G5 marketplace registration check** | P1-4 | Every `*/.claude-plugin/plugin.json` has a marketplace.json entry (or an explicit `unlisted` allowlist). |
| **G6 description linter upgrade** | P1-8 | Extend `skill_description_validator.py`: hard-fail >1024 chars, warn missing trigger phrasing, apply to agents too; fix `templates/agent-template.md` so new agents start compliant. |
| **G7 model-name freshness** | stale GPT-4/claude-3 era refs | Regex deny-list for retired model identifiers outside historical-context sentences (2 SKILL.md + 2 scripts today). |
| **G8 argparse contract** | P0-11 | Every `scripts/*.py` must exit 0 on `--help` within 5s unless listed in a by-design exceptions file (hooks, fixed evaluators). |
| **G9 meta-tool hygiene** | P1-9 | `generate-docs.py --help` must be side-effect-free; sync scripts/convert.sh/compileall enumerate domains from the tree, not a hardcoded list. |
---
## 6. Suggested execution order (follow-up PRs)
1. **PR-1 (P0 batch, ~1 day):** fix the 11 P0 defects. Highest stakes first: FDA/QMSR, ALARP, AI-Act sample, financial-analyst JSON shape, deal-desk formula, corrupted code literals, Atlassian tool-name appendix, webinar argparse, markdown-html orchestrator status text, retire command-guide + google-workspace-cli (or re-verify the CLI exists).
2. **PR-2 (path sweep):** G1 linter + one mechanical fix-all for the `skills/` segment; fix c-level agent KB filenames, research `office/validate.py` (write it or drop the step), email agent paths.
3. **PR-3 (registries):** G3 counter derivation + marketplace registration of the 11 plugins + c-level role-registry update (6 files) + marketing context-file unification.
4. **PR-4 (wiring):** orphan-script A3 pass per domain report lists; trigger-description batch (79 skills, 77 agents) using each report's per-skill suggested descriptions.
5. **PR-5 (CI):** land gates G2, G4–G9; remove 35 .zip archives; fix meta-tooling.
6. **PR-6+ (content):** the 16 REWRITE skills, then OPTIMIZE queue per domain, using the per-skill "Verify (definition of done)" blocks as acceptance criteria.
---
## 7. Where the per-skill verification criteria live
The user-requested **customized verification loops and validation criteria for every skill, plugin, agent, and command** are in the domain reports: every OPTIMIZE/REWRITE/CUT entry carries a "Verify (definition of done)" block of 2–4 executable checks, and every KEEP skill has a one-line verification contract in its report's "KEEP-verdict verification criteria" section. Those blocks are the input for gate G2.
audit/newgen-2026-06/RUBRIC.md
+55
View File
@@ -0,0 +1,55 @@
# New-Generation Model Optimization Rubric (v1)
Audit date: 2026-06-10 · Branch: `claude/skills-plugins-audit-vrttx1`
This rubric defines what "optimized for new-generation models" means for every artifact
type in this repository. New-gen frontier models (Claude Fable/Opus 4.x class) differ from
the models many of these skills were written for: they need **less hand-holding, more
context economy, and machine-checkable verification**. A skill earns its context window
or it gets cut.
## A. SKILL.md — 7 dimensions
| # | Dimension | What PASS looks like |
|---|-----------|----------------------|
| A1 | **Trigger quality** | Frontmatter `description` states what the skill does AND when to fire, in third person, < 1024 chars, with concrete trigger phrases ("Use when…", example user requests). Never just the skill name. |
| A2 | **Context economy** | Body is instruction-dense. Progressive disclosure: SKILL.md holds the workflow + decision rules; deep knowledge lives in `references/` and is loaded on demand. No "You are an expert…" filler, no restating what a frontier model already knows (generic advice = dead weight). |
| A3 | **Tool wiring** | Every referenced script exists, has exact CLI invocations in SKILL.md, and its output is consumed by a named next step. No orphan scripts, no phantom paths. |
| A4 | **Verification loop** | The workflow ends with a check the model can execute: run a script and assert exit code/output shape, validate against an explicit checklist, or hit a refusal gate. Skills without one get a *proposed* custom loop in this audit. |
| A5 | **Real-world expertise** | A practitioner would recognize domain mastery: named frameworks with thresholds, formulas, regulatory citations, decision trees — not "communicate clearly with stakeholders". |
| A6 | **Freshness** | No stale model names (claude-3-x, GPT-3.5-era), dead prices, or 2024-isms presented as current. |
| A7 | **No filler files** | Every file in the package earns its place. References cite sources and say something non-obvious. Assets are usable, not shells. No duplicated boilerplate. |
Verdicts: **KEEP** (ship as is) · **OPTIMIZE** (targeted edits, listed) · **REWRITE** (structure salvageable, content not) · **CUT-OR-MERGE** (does not earn its context).
## B. Agents (`agents/*.md`)
- B1 Frontmatter: `name`, `description` with trigger phrasing ("Use when…" / "Use PROACTIVELY…"), minimal `tools` list.
- B2 Differentiation: the system prompt could not be swapped with a sibling agent's without someone noticing. Personas must change behavior, not adjectives.
- B3 No placeholder/boilerplate body.
## C. Slash commands (`commands/*.md`)
- C1 Frontmatter description present and accurate.
- C2 `$ARGUMENTS` / argument-hint handled.
- C3 The command does something a bare prompt could not (orchestrates tools, enforces gates). Otherwise: candidate for merge or cut.
## D. Scripts (`scripts/*.py`)
- D1 `--help` exits 0 (verified repo-wide this audit: 583/593 pass).
- D2 Stdlib-only, no LLM calls, deterministic.
- D3 Output is machine-parseable where a workflow consumes it (JSON mode).
- D4 By-design exceptions (hooks reading stdin, fixed-contract evaluators) documented where they live.
## E. Plugins (`.claude-plugin/plugin.json`)
- E1 Schema valid (verified repo-wide: all pass `check_plugin_json.py --all`).
- E2 Description matches actual contents (pod/skill counts drift).
- E3 Version coherent with marketplace.json.
## Custom verification criteria — the contract
For **every skill** audited, the domain report includes 2–4 *executable* "definition of
done" checks specific to that skill (e.g. "`python3 scripts/x.py --sample` exits 0 and
emits JSON with keys `verdict`, `score`; verdict ∈ {GO, NO-GO}"). These are the
guardrails future optimization PRs must keep green.
audit/newgen-2026-06/bizops-commercial-finance.md
+133
View File
@@ -0,0 +1,133 @@
# Domain audit: business-operations/ + commercial/ + finance/ + business-growth/ — new-gen model optimization
Audited: 2026-06-10 · Skills: 24 · Agents: 2 · Commands: 17 (+2 root finance commands) · Plugins: 5
## Scorecard
| Skill | Verdict | Top issue |
|---|---|---|
| business-operations/business-operations-skills | KEEP | — (orchestrator routing is real: signal table + 2-signal threshold + no-silent-chain) |
| business-operations/process-mapper | KEEP | — (deterministic VA% bands, 3 detection rules, profiles) |
| business-operations/vendor-management | OPTIMIZE | Frontmatter description 1,106 chars (> 1024 A1 limit) |
| business-operations/capacity-planner | KEEP | — (Erlang-C, P50/P90/P99, manager-trigger; best forcing-question library in scope) |
| business-operations/internal-comms | OPTIMIZE | Description 1,284 chars (> 1024) |
| business-operations/knowledge-ops | OPTIMIZE | Description 1,314 chars (> 1024) — the actual cause of its 2/6 repo-checklist infamy; content itself is strong |
| business-operations/procurement-optimizer | OPTIMIZE | Description 1,266 chars (> 1024) |
| commercial/commercial-skills | KEEP | — (orchestrator; 7-lane signal table, depth-first chaining gates) |
| commercial/pricing-strategist | KEEP | — (model+range hard rule operationalized in workflow Step 5 + anti-pattern #1) |
| commercial/deal-desk | OPTIMIZE | Margin math is internally contradictory (SKILL.md says 37.5% loss; script computes 24-pt / 30% via a different formula) |
| commercial/partnerships-architect | KEEP | — (deterministic tier floors, kill-criteria mandate) |
| commercial/channel-economics | OPTIMIZE | Description 1,178 chars (> 1024) |
| commercial/commercial-policy | KEEP | — (4-dim matrix, 10-rule linter, precedent-risk flag verified) |
| commercial/rfp-responder | KEEP | — (GAP-never-invent rule, no-bid < 20% threshold; winrate sample verified) |
| commercial/commercial-forecaster | KEEP | — (assumption block verified NON-OPTIONAL in script output) |
| finance/finance-skills | CUT-OR-MERGE | 55-line plugin README posing as a skill; broken paths; no trigger description |
| finance/financial-analyst | OPTIMIZE | All 4 scripts silently fail (exit 0, zeros) against their own bundled sample — schema mismatch |
| finance/saas-metrics-coach | KEEP | — (benchmark tables with thresholds, strict output contract, status labels) |
| finance/business-investment-advisor | OPTIMIZE | Nested plugin unregistered in marketplace; manifest description truncated mid-word |
| business-growth/business-growth-skills | CUT-OR-MERGE | References phantom "BizDev-toolkit" skill; broken quick-start paths |
| business-growth/contract-and-proposal-writer | OPTIMIZE | 423-line SKILL.md with full contract templates inline; no scripts/references/assets at all |
| business-growth/customer-success-manager | KEEP | — (weighted scorers with explicit weights/thresholds, segment-aware) |
| business-growth/revenue-operations | KEEP | — (formula+threshold tables, CRM cross-check verification steps) |
| business-growth/sales-engineer | KEEP | — (executable validation checkpoints between phases; strongest A4 in scope) |
**Counts: 14 KEEP · 8 OPTIMIZE · 0 REWRITE · 2 CUT-OR-MERGE**
## Domain-level findings
1. **The v2.8.0 claims hold.** Both orchestrators (`business-operations-skills`, `commercial-skills`) have `context: fork` in frontmatter and real routing logic — signal tables, a 2-signal confidence threshold, single-question fallback with recommended answer, and explicit "never silently chain" gates. Every one of the 13 bizops/commercial sub-skills ships a 5-8 question forcing-question library with per-question recommended answer + canon citation. This is not vague prose.
2. **Hard rules are operationalized, not just claimed.** Verified by execution: `deal_scorer.py --sample` emits an approver chain and "2 critical signal(s) detected; cannot APPROVE" (never auto-approves); `bookings_forecaster.py --sample` emits an "Assumption block (NON-OPTIONAL)" section; pricing-strategist's workflow Step 5 and anti-pattern #1 enforce model+range-never-a-number; vendor/procurement outputs are framed as "inputs to a human decision" with refusal logic (single-source tier-1 consolidation refused without break-glass).
3. **Systemic A1 failure in the v2.8.0 batch: 5 of 15 frontmatter descriptions exceed the 1,024-char limit** (knowledge-ops 1,314 · internal-comms 1,284 · procurement-optimizer 1,266 · channel-economics 1,178 · vendor-management 1,106). They cram "Distinct from" and tool inventories into the trigger field. This — not content quality — is what tanked knowledge-ops/procurement-optimizer on the repo's own checklist. One-pass fix: move everything after the trigger sentence into the body.
4. **Systemic A3 gap in the v2.8.0 batch: almost no fenced CLI examples.** 8 of 13 sub-skills score "0 code blocks" on the repo checklist; invocations live in prose/tables. deal-desk and commercial-policy (which have "Quick examples" sections) show the right pattern.
5. **finance/ and business-growth/ are a different generation.** No forcing questions, no profiles, no named-owner routing, meta-skills (`finance-skills`, `business-growth-skills`) that are plugin READMEs with broken paths. The per-skill content (saas-metrics-coach, customer-success-manager, revenue-operations, sales-engineer) still earns KEEP on calibrated thresholds, but the wrappers are dead weight.
6. **One genuinely broken tool chain: financial-analyst.** Its `assets/sample_financial_data.json` nests data under per-tool keys (`ratio_analysis`, `dcf_valuation`, `budget_variance`, `forecast`) while all four scripts read top-level keys. Result: every documented quick-start command "succeeds" (exit 0) while emitting all-zero ratios, "Error: Historical revenue data is required" (still exit 0), `Total Items: 0`, and $0.00 forecasts. The repo-wide `--help` smoke test cannot catch this class of failure.
7. **Domain-math inconsistency in deal-desk.** SKILL.md (citing `discount_economics.md`) says "a 30% discount on an 80% gross-margin product loses 37.5% of margin, not 30%" — correct under fixed COGS: margin dollars fall 30/80 = 37.5%. But `deal_scorer.py` and `discount_economics.md` actually use `net_margin = G − D·(G/100)` (proportional COGS) → 24-pt loss / 30% relative. The script's own docstring writes the correct `(G−D)/(1−D/100)` formulation, then discards it. Pick one model; the scorer's margin dimension (weight 0.30) currently understates discount damage.
## Per-skill findings
### business-operations/vendor-management — OPTIMIZE
- Issues: (1) description 1,106 chars > 1024 — trim to trigger sentence, move "Ships 3 tools…" and "Distinct from…" into the body; (2) no fenced CLI examples (checklist rule 5); (3) SKILL.md 170 lines — body content fine, but Steps 2-4 partially duplicate script-level docs.
- Verify: `python3 -c "import re;t=open('business-operations/skills/vendor-management/SKILL.md').read();d=re.search(r'description:(.*?)\n\w+:',t,re.S).group(1);assert len(d.strip())<1024"` · `python3 skills/vendor-management/scripts/vendor_scorer.py --sample` exits 0 with KEEP/REVIEW/REPLACE verdicts · `--profile healthcare` output differs from `--profile saas` · SKILL.md contains ≥ 1 fenced ```bash block.
### business-operations/internal-comms — OPTIMIZE
- Issues: (1) description 1,284 chars > 1024; (2) zero fenced CLI examples; (3) "Distinct from" appears in both description and body (duplicated context cost).
- Verify: description < 1024 chars · all 3 scripts pass `--sample` exit 0 · `change_announcement_builder.py` still rejects "exciting news" tone on `disruptive` magnitude (magnitude/tone validation preserved) · ≥ 1 fenced code block in SKILL.md.
### business-operations/knowledge-ops — OPTIMIZE
- Issues: (1) description 1,314 chars > 1024 — worst in repo, and the real driver of its 2/6 checklist score; (2) zero fenced CLI examples; (3) "a procurement tool sunset in 2024" tripped the time-sensitivity rule — rephrase relatively; (4) content (5W2H validator thresholds, SAFE ≥ 80 / NOT-SAFE < 60 bands, staleness×inbound-links ranking) is genuinely strong — do not rewrite.
- Verify: description < 1024 chars · `kb_ingester.py --sample` exits 0 and reports orphan/stale/glossary-drift counts on the synthetic 8-page vault · `runbook_validator.py --sample` returns NOT-SAFE on the deliberately-broken runbook · repo checklist (`skill_review_checklist_runner.py <folder>`) reaches ≥ 4/6.
### business-operations/procurement-optimizer — OPTIMIZE
- Issues: (1) description 1,266 chars > 1024; (2) zero fenced CLI examples; (3) SKILL.md 167 lines with tool inventory repeated 3× (description, Workflow, Scripts table).
- Verify: description < 1024 chars · `supplier_consolidation.py --sample` exits 0 and still emits the "DO NOT CONSOLIDATE — tier-1 cluster, no break-glass" refusal on a tier-1 cluster without break-glass flag · `spend_categorizer.py --sample --profile enterprise` differs from `--profile tech-startup`.
### commercial/deal-desk — OPTIMIZE
- Issues: (1) margin-formula contradiction: SKILL.md claims 37.5% margin loss (fixed-COGS, correct), `deal_scorer.py:123-133` + `references/discount_economics.md` compute `G − D·G/100` = 24 pts (proportional-COGS); the docstring names the correct `(G−D)/(1−D/100)` formula then ignores it — reconcile to one model across all three files; (2) discount_economics.md "Why the conventional shorthand is wrong" section is itself wrong under the standard fixed-COGS assumption.
- Verify: `deal_scorer.py --sample` exits 0, verdict DECLINE, output contains "cannot APPROVE" and a ≥ 4-hop approver chain · SKILL.md margin example, `discount_economics.md` worked table, and `score_margin()` produce the same number for (G=80, D=30) · `--profile services` composite differs from `--profile saas` · `terms_redliner.py --sample` flags uncapped indemnity as CRITICAL.
### commercial/channel-economics — OPTIMIZE
- Issues: (1) description 1,178 chars > 1024 — it embeds four "Not X (that's Y)" disambiguations plus a keyword list; (2) zero fenced CLI examples.
- Verify: description < 1024 chars · all 3 scripts pass `--sample` exit 0 · `channel_roi_analyzer.py --sample` emits one of DOUBLE-DOWN/MAINTAIN/DEFUND/EXIT per channel · cost-to-serve output contains both per-deal and per-$-ARR lines.
### finance/finance-skills — CUT-OR-MERGE
- Issues: (1) it is a plugin README, not a skill — no workflow, no decision rules, no trigger phrasing ("Financial analyst agent skill and plugin for Claude Code, Codex…"); (2) quick-start path `finance/financial-analyst/SKILL.md` doesn't exist (actual: `finance/skills/financial-analyst/`); (3) wholly duplicates plugin.json + finance/CLAUDE.md. Merge useful lines into README.md/plugin.json and delete, or rebuild as a real router (the bizops/commercial orchestrator pattern exists to copy).
- Verify (if merged): `finance/skills/finance-skills/` removed AND `finance/.claude-plugin/plugin.json` `skills` array updated AND `check_plugin_json.py --all` passes · no references to `finance/financial-analyst/` (without `/skills/`) remain: `grep -r "finance/financial-analyst" finance/ | grep -v skills/` returns nothing.
### finance/financial-analyst — OPTIMIZE
- Issues: (1) **broken tool wiring**: all 4 scripts read top-level keys (`income_statement`, …) while `assets/sample_financial_data.json` nests them under `ratio_analysis`/`dcf_valuation`/`budget_variance`/`forecast` — every documented quick-start emits zeros; (2) error masking: `dcf_valuation.py` prints "Error: Historical revenue data is required" and exits 0; (3) `references/financial-ratios-guide.md` cites zero sources (A7); (4) Phase 1/5 of the workflow is filler a frontier model doesn't need ("Define analysis objectives and stakeholder requirements").
- Verify: `ratio_calculator.py assets/sample_financial_data.json` produces zero "Insufficient data" lines and a nonzero Gross Margin · `dcf_valuation.py <bad input>` exits nonzero · `budget_variance_analyzer.py assets/sample_financial_data.json` reports Total Items > 0 · `forecast_builder.py` base-case revenue > $0 on the bundled sample.
### finance/business-investment-advisor — OPTIMIZE
- Issues: (1) nested plugin `finance/business-investment-advisor/.claude-plugin/plugin.json` is not registered in marketplace.json — either register it or fold the skill into the finance plugin's `skills` array and delete the nested manifest; (2) that manifest's description is truncated mid-word ("Also use f"); (3) prompt-only skill claiming "show all math" with no deterministic tool — acceptable for an advisor, but the IRR/NPV sections restate model-known formulas (A2); the rubric + proactive-triggers + anti-pattern table are the parts that earn context — trim the formula restatements.
- Verify: skill is reachable via exactly one registered plugin (`python3 scripts/check_plugin_json.py --all` passes; marketplace lookup finds it) · manifest description is a complete sentence < 1024 chars · SKILL.md ≤ ~150 lines after trimming formula primers.
### business-growth/business-growth-skills — CUT-OR-MERGE
- Issues: (1) names a fifth skill "BizDev-toolkit" that does not exist anywhere in the repo; (2) quick-start path `business-growth/customer-success-manager/SKILL.md` is wrong (actual: `business-growth/skills/customer-success-manager/`); (3) body says "4 production-ready skills", description says 5 — neither matches a real router; (4) duplicates plugin.json + CLAUDE.md. Same disposition as finance-skills: delete-and-merge, or rebuild on the bizops orchestrator pattern.
- Verify (if merged): folder removed, plugin.json skills path still valid, `grep -ri "bizdev-toolkit" business-growth/` returns nothing, `check_plugin_json.py --all` passes.
### business-growth/contract-and-proposal-writer — OPTIMIZE
- Issues: (1) 423-line SKILL.md with three full contract templates + a GDPR DPA block inline — move Templates A/B/C and the DPA block to `assets/` and keep selection logic + jurisdiction notes + pitfalls in SKILL.md (progressive disclosure, A2); (2) only single-file skill in scope — no references/assets despite being template-heavy by nature; (3) static legal claims (§126 BGB, §74 HGB, "post-Brexit") carry freshness risk with no last-reviewed marker; add one; (4) overlaps `commercial/rfp-responder` and `c-level-advisor/general-counsel-advisor` — the existing scope sentence is good, keep it.
- Verify: SKILL.md ≤ ~150 lines · `assets/` contains ≥ 4 template files referenced by name from SKILL.md · description still trigger-phrased ("Use when drafting…") · a "last legal review" date line exists.
## KEEP-verdict verification criteria
- **business-operations-skills**: frontmatter retains `context: fork`; signal table lists exactly 6 lanes matching plugin.json skill paths; "Do NOT chain silently" and ≤ 200-word digest rules present.
- **process-mapper**: 3 scripts pass `--sample` exit 0; `cycle_time_analyzer` emits VA% verdict ∈ {HEALTHY, TYPICAL, WASTE-HEAVY} with 25%/10% bounds; `bottleneck_detector.py --profile healthcare` ≠ `--profile saas` output.
- **capacity-planner**: `capacity_modeler.py --sample` exits 0 with risk band ∈ {SAFE, WATCH, AT_RISK, CRITICAL} and a P50/P90/P99 breach table; `hiring_sequencer` triggers a manager hire when span crosses 7; 7 forcing questions remain canon-cited.
- **commercial-skills**: `context: fork` retained; 7-lane signal table matches the 7 sub-skill folders; anti-pattern list keeps "recommend a range + model" and "never auto-approve" lines.
- **pricing-strategist**: `wtp_analyzer.py --sample` exits 0, emits OPP/IDP/PMC/PME + a sub-100-N sample-size warning; `packaging_designer.py --sample` flags ≥ 1 anti-pattern; SKILL.md anti-pattern #1 ("Recommending a specific number") intact.
- **partnerships-architect**: `partner_tier_classifier.py --sample` exits 0; STRATEGIC floor still requires named_accounts ≥ 5 AND multi-year commit AND dedicated resources; kill-criteria mandate in Assumptions.
- **commercial-policy**: `policy_linter.py --sample` exits FAIL-by-design with 4 BLOCKERs; `exception_router.py --sample` routes the 42% exception with ≥ 3 compensating commitments; precedent-risk (3+ similar exceptions) flag preserved.
- **rfp-responder**: `winrate_predictor.py --sample` exits 0 with estimate + confidence band + verdict ∈ {BID, PARTNER-BID, NO-BID}; < 20% auto-no-bid threshold intact; "never invents claims" GAP rule in both description and Step 2.
- **commercial-forecaster**: `bookings_forecaster.py --sample` output contains "Assumption block (NON-OPTIONAL"; three tiers (commit/best-case/pipe-only) emitted; CoV bands (10/25/50%) in `funnel_confidence_scorer` unchanged.
- **finance/saas-metrics-coach**: `metrics_calculator.py --mrr 50000 --customers 100 --churned 5 --json` exits 0 with `_missing` array; quick-ratio bands (<1 CRITICAL, >4 EXCELLENT) match references/benchmarks.md; output-format template (Metrics at a Glance → 90-Day Focus) preserved.
- **customer-success-manager**: all 3 scripts exit 0 against `assets/sample_customer_data.json` (its sample actually works — unlike financial-analyst's); dimension weights sum to 100% in both SKILL.md tables and scripts; Green/Yellow/Red bounds 75/50 unchanged.
- **revenue-operations**: 3 scripts exit 0 against bundled samples; coverage target 3-4x, MAPE rating table, and Rule-of-40 thresholds present in both SKILL.md and script output.
- **sales-engineer**: the inline python validation one-liners in Phases 1/3/4 execute without KeyError against `--format json` output of the corresponding scripts (these are the skill's A4 backbone — keep them runnable).
## Agents
2 agents in scope (`business-operations/agents/cs-bizops-orchestrator.md`, `commercial/agents/cs-commercial-orchestrator.md`); finance/ and business-growth/ ship none.
- **B1**: both have name/description/tools/model. Descriptions are persona-stated rather than "Use when…"-phrased — minor A1-style gap; routing context is otherwise clear. Both pin `model: sonnet` — verify that's intended for orchestration-heavy work on new-gen defaults.
- **B2**: genuinely differentiated — different signature questions ("Where does the work spend most of its time waiting?" vs "What's the margin at full discount?"), different lane tables, different hard-output contracts (named approver / model+range / disclosed assumption). Not swappable.
- **B3**: no boilerplate. One staleness bug in both: the "Available commands" sections still annotate 9 commands with "(Sprint 2)" although all shipped — delete the annotations.
- Gap: business-growth and finance rely on root-level `/saas-health`, `/financial-health` and no persona agent; acceptable for legacy skills, but if the meta-skills are rebuilt as routers, add agents then — not before.
## Commands
17 in scope (8 bizops, 9 commercial) + 2 root finance commands (`/saas-health`, `/financial-health`).
- **C1/C2**: all sampled commands have accurate frontmatter descriptions and `argument-hint`, and interpolate `$ARGUMENTS`.
- **C3**: routers (`/cs:bizops`, `/cs:commercial`) and grills (`/cs:grill-bizops`, `/cs:grill-commercial`) clearly orchestrate (signal scoring, one-question discipline, refusal-to-route gates) — pass. Per-skill commands (`/cs:vendor-review`, `/cs:deal-review`, `/cs:knowledge-ops`, etc.) mostly restate their SKILL.md's tool table + "Distinct from" section; they pass C3 only because they name exact tools/profiles/verdicts. If context budget ever matters, these 13 per-skill commands are the first merge candidates into their routers — but no action required now.
- `/saas-health` and `/financial-health` are thin wrappers over the scripts; `/financial-health` inherits the financial-analyst sample-schema bug (its documented `ratios <data.json>` path silently zeroes). Fix rides on the financial-analyst OPTIMIZE.
- business-growth/ has zero commands — its 4 real skills are invocable only by description-matching. Acceptable; note for any future refresh.
## Plugin manifests
All 5 schema-valid (repo-wide E1 pass confirmed); versions coherent at 2.9.0 where registered. E2 drift everywhere:
1. **business-operations**: claims "24+ reference docs each citing ≥7 authoritative sources" — actual count is 18 (6 sub-skills × 3). Fix the number.
2. **commercial**: claims "28+ reference docs" — actual is 21 (7 × 3). Fix the number.
3. **finance**: description counts business-investment-advisor among its "3 finance skills", but `"skills": ["./skills"]` excludes it (it lives at `finance/business-investment-advisor/`, outside the path). Either move the skill under `finance/skills/` or correct the description.
4. **business-investment-advisor** (nested): description truncated mid-word ("…Also use f"); not registered in `.claude-plugin/marketplace.json` — it is currently undiscoverable as a plugin. Register or fold into finance-skills.
5. **business-growth**: description enumerates "BizDev-toolkit", a skill that does not exist; "5 business & growth skills" counts the meta-README skill. Correct to the 4 real skills (or 5 only after the meta-skill is rebuilt as a real router).
audit/newgen-2026-06/c-level-advisor.md
+281
View File
@@ -0,0 +1,281 @@
# Domain audit: c-level-advisor/ — new-gen model optimization
Audited: 2026-06-10 · Unique skills: 61 (of 66 SKILL.md files incl. dual-published) · Agents: 14 (13 cs-* in c-level-agents/agents/ + devils-advocate; cs-ceo/cs-cto live outside the folder in /agents/c-level/) · Plugins: 8
## Scorecard
| Skill | Verdict | Top issue |
|---|---|---|
| **skills/ (main bundle, 33)** | | |
| agent-protocol | OPTIMIZE | Valid-roles list frozen at 9 roles; 5 newer roles (GC/CDO/CAIO/CCO/VPE) can't be invoked per protocol |
| board-deck-builder | OPTIMIZE | Phantom `/board-deck` command in Quick Start |
| board-meeting | OPTIMIZE | Role tables omit 5 newer roles; `/cs:board` vs `/cs:boardroom` naming clash |
| c-level-skills | CUT-OR-MERGE | Bundle README posing as a skill; contradicts cs-onboard and board-meeting on protocol details |
| ceo-advisor | KEEP | ~30 lines of shared boilerplate (Communication/Context Integration) duplicated across all role skills |
| cfo-advisor | KEEP | — |
| change-management | KEEP | — |
| chief-ai-officer-advisor | KEEP | A6 watch: hardcoded 2026 API/GPU pricing will rot |
| chief-customer-officer-advisor | KEEP | — |
| chief-data-officer-advisor | KEEP | — |
| chief-of-staff | OPTIMIZE | "28 skills" stale (33); routing matrix omits 5 roles; 3rd divergent decision-log path |
| chro-advisor | KEEP | — |
| ciso-advisor | KEEP | — |
| cmo-advisor | KEEP | — |
| company-os | KEEP | — |
| competitive-intel | OPTIMIZE | 5 phantom `/ci:*` commands in Quick Start |
| context-engine | KEEP | — |
| coo-advisor | KEEP | — |
| cpo-advisor | KEEP | — |
| cro-advisor | KEEP | — |
| cs-onboard | OPTIMIZE | Conflicts with c-level-agents `/cs:onboard` (7-dimension vs 12-question interview, same output file) |
| cto-advisor | KEEP | — |
| culture-architect | KEEP | — |
| decision-logger | KEEP | Memory path conflict with `/cs:decide` (flagged there) |
| founder-coach | KEEP | — |
| general-counsel-advisor | KEEP | — |
| internal-narrative | KEEP | — |
| intl-expansion | OPTIMIZE | Thin; no tool; no verification loop |
| ma-playbook | OPTIMIZE | Thinnest skill in domain; valuation numbers unsourced; no tool |
| org-health-diagnostic | KEEP | — |
| scenario-war-room | KEEP | Phantom `/war-room` invocation (minor) |
| strategic-alignment | KEEP | — |
| vpe-advisor | KEEP (dual-published) | Workflow CLI paths inconsistent with Quick Start paths |
| **executive-mentor/skills/ (6)** | | |
| executive-mentor | KEEP | — |
| challenge | KEEP | — |
| board-prep | KEEP | — |
| hard-call | OPTIMIZE | Placeholder description (A1 fail) |
| postmortem | OPTIMIZE | Placeholder description (A1 fail) |
| stress-test | OPTIMIZE | Placeholder description (A1 fail) |
| **c-level-agents/skills/ (22)** | | |
| c-level-agents (overview) | OPTIMIZE | Frontmatter says 8 agents / 17 commands; reality is 13 / 21 |
| founder-mode | OPTIMIZE | Routing table omits CDO/CAIO/CCO/VPE — auto-router can't reach 4 of 13 advisors |
| office-hours | KEEP | — |
| onboard | OPTIMIZE | Second, divergent founder interview writing the same `~/.claude/company-context.md` |
| brief | OPTIMIZE | Affected-roles checklist omits 5 newer advisors |
| boardroom | KEEP | — |
| decide | OPTIMIZE | Writes `~/.claude/decisions/` while decision-logger skill specifies `memory/board-meetings/` |
| execute | KEEP | — |
| post-mortem | KEEP | — |
| freeze | KEEP | `/cs:unfreeze` has no skill file (handled in-file; minor) |
| cross-eval | KEEP | — |
| cfo-review | KEEP | — |
| cmo-review | KEEP | — |
| cpo-review | KEEP | — |
| cro-review | KEEP | — |
| cto-review | KEEP | — |
| ciso-review | KEEP | — |
| gc-review | KEEP | — |
| cdo-review | OPTIMIZE | Routes to phantom `/cs:chro-review` |
| caio-review | OPTIMIZE | Routes to phantom `/cs:chro-review` |
| cco-review | OPTIMIZE | Routes to phantom `/cs:chro-review` |
| vpe-review | OPTIMIZE | Routes to phantom `/cs:chro-review` |
| **Dual-published standalone copies** | (counted above) | chief-ai-officer-advisor, chief-customer-officer-advisor, chief-data-officer-advisor, general-counsel-advisor, vpe-advisor |
**Totals: KEEP 40 · OPTIMIZE 20 · REWRITE 0 · CUT-OR-MERGE 1**
## Domain-level findings
### 1. Dual-publication map (5 pairs, zero drift today — but no guard)
Each of these exists twice, byte-identical (verified with `diff -rq` across SKILL.md + all references + all scripts):
| Bundle copy (`c-level-advisor/skills/<x>/`) | Standalone copy (`c-level-advisor/<x>/skills/<x>/`) | Drifted? |
|---|---|---|
| chief-ai-officer-advisor | chief-ai-officer-advisor | No |
| chief-customer-officer-advisor | chief-customer-officer-advisor | No |
| chief-data-officer-advisor | chief-data-officer-advisor | No |
| general-counsel-advisor | general-counsel-advisor | No |
| vpe-advisor | vpe-advisor | No |
The duplication is intentional (standalone-installable plugin AND bundled in c-level-skills) but there is **no sync script, no CI check, and no comment in either copy declaring the other copy exists**. Any future edit to one side silently forks the skill. **Recommendation:** add a `scripts/check_dual_published.py` (or extend ci-quality-gate) that diffs the 5 pairs and fails on divergence; alternatively replace standalone copies with a build step. This is the single highest-leverage guard for the domain. Drift status: GREEN today, unprotected.
### 2. Role-registry drift — the domain's 9-role core never learned about its 5 newest roles
The domain grew from 9 C-roles to 14 (GC v2.5.1, CDO v2.5.2, CAIO v2.5.3, CCO v2.5.4, VPE v2.5.5), but the orchestration core was never updated:
- `agent-protocol/SKILL.md` — "Valid roles: ceo, cfo, cro, cmo, cpo, cto, chro, coo, ciso". Per the protocol's own hard rules, `[INVOKE:gc|...]` etc. is undefined.
- `chief-of-staff/SKILL.md` — routing matrix and "routes to 28 skills total" (now 33) omit the 5 roles entirely.
- `board-meeting/SKILL.md` — Phase 1 role-activation table and Phase 2 ordering list only the original 9.
- `c-level-agents/skills/founder-mode/SKILL.md` — keyword routing table has no rows for data/AI/customer/VPE topics; "retention dropped" routes to CRO, never CCO.
- `c-level-agents/skills/brief/SKILL.md` — affected-roles checklist stops at cs-chief-of-staff.
- `c-level-agents/skills/c-level-agents/SKILL.md` frontmatter — "8 cs-* agents… 17 /cs:* commands" vs actual 13 / 21.
Fix once, in all six files, in one PR.
### 3. Three competing decision-memory architectures
- `decision-logger` skill: `memory/board-meetings/decisions.md` (Layer 2) + `YYYY-MM-DD-raw.md` (Layer 1)
- `chief-of-staff` skill: `~/.claude/decision-log.md`
- `/cs:decide`: `~/.claude/decisions/approved/` + `~/.claude/decisions/raw/`
All three claim to be "the" two-layer memory. An agent following decision-logger will never find decisions written by `/cs:decide`. Pick one canonical layout (the `~/.claude/decisions/` form is most portable) and update the other two.
### 4. Two competing founder-onboarding interviews
`cs-onboard` (7 dimensions, ~45 min, conversational probes) and `c-level-agents/onboard` (12 structured questions) both write `~/.claude/company-context.md` with **different schemas**; `c-level-skills/SKILL.md` describes a third, 7-question variant that writes to "the project root". The context-engine "Required Context Fields" match only the cs-onboard schema. Reconcile to one interview + one schema.
### 5. Phantom slash commands
Referenced but existing nowhere in `commands/` nor as plugin command/skill files: `/board-deck`, `/war-room`, `/health`, `/health:dimension`, `/ci:landscape|battlecard|winloss|update|map`, `/cs:chro-review`, `/cs:coo-review` (routed-to from 4+ review skills), `/cs:board` + `/cs:decisions` + `/cs:review` + `/cs:setup` + `/cs:update` (pre-date the c-level-agents `/cs:*` namespace and now collide with it). Either create thin command files or rewrite the Quick Starts as natural-language triggers.
### 6. Agent reference-file hallucinations (see Agents section)
7 of 13 cs-* agents cite knowledge-base filenames that do not exist on disk — apparently written from memory of what the references *should* be called. New-gen models will attempt to Read these paths and fail.
### 7. Shared boilerplate tax
Every role-advisor SKILL.md carries the identical ~25-line Communication / Context Integration / Internal Quality Loop block that `agent-protocol/SKILL.md` already owns. Across ~15 skills that's ~375 lines of duplicated context. A one-line pointer ("Output passes the Internal Quality Loop — see agent-protocol/SKILL.md") would reclaim it. Also: `Keywords` sections (10–40 terms each) are dead weight for new-gen trigger matching since the frontmatter description already carries triggers.
### 8. Repo hygiene
- `ceo-advisor.zip` (32K) and `cto-advisor.zip` (28K) are stray build artifacts at the domain root — delete.
- `c_level_leadership_skills_overview.md` at domain root duplicates CLAUDE.md content.
- `c-level-advisor/CLAUDE.md` says "Skills Deployed: 33 … + 21 /cs:* sub-skills" but also "28 skills" in the architecture intro of chief-of-staff; CLAUDE.md itself says "13 cs-* persona agents" correctly but root repo CLAUDE.md says "51+ agents (cs-* + 7 personas)" — counts drift in 3 places.
## Per-skill findings
### c-level-skills (skills/c-level-skills/)
- **Verdict: CUT-OR-MERGE**
- Issues: (1) It's the bundle's README wearing a SKILL.md frontmatter — `name: "c-level-advisor"` doesn't even match its folder. (2) Describes `/cs:setup` as a 7-question form writing company-context.md "to the project root", contradicting cs-onboard (7 dimensions, `~/.claude/`) and c-level-agents/onboard (12 questions). (3) Describes `/cs:board` as 3 phases; board-meeting skill defines 6. (4) Stale counts (28 skills, 25 tools). (5) As a trigger surface it competes with chief-of-staff for the same routing job.
- Verify: `grep -c "Phase" skills/c-level-skills/SKILL.md` no longer describes a board protocol that disagrees with `board-meeting/SKILL.md`; content merged into README.md or CLAUDE.md; `find c-level-advisor/skills -name SKILL.md | wc -l` drops by 1 (or file becomes a pure router stub < 40 lines).
### agent-protocol
- **Verdict: OPTIMIZE**
- Issues: (1) Valid-roles list omits gc/cdo/caio/cco/vpe. (2) Peer-verification table has no rows for legal/data/AI/customer claims. (3) At 418 lines it carries the Communication Standard that 15 sibling skills then duplicate — the duplication should collapse toward this file.
- Verify: `grep -E "gc|cdo|caio|cco|vpe" skills/agent-protocol/SKILL.md` returns the invocation registry rows; the 5 newer role SKILL.mds still render the `[INVOKE:role|...]` examples without contradiction; sibling role skills reference (not restate) the quality loop.
### chief-of-staff
- **Verdict: OPTIMIZE**
- Issues: (1) "routes to 28 skills total" — 33 exist. (2) Routing matrix has no rows for legal, data strategy, AI strategy, customer/retention, eng-delivery topics. (3) Decision log path `~/.claude/decision-log.md` is a third memory location (see domain finding 3). (4) References `references/routing-matrix.md` — confirm it covers all 14 roles too.
- Verify: routing matrix includes GC/CDO/CAIO/CCO/VPE rows; `grep "28 skills" SKILL.md` → no hits; decision-log path identical to decision-logger's canonical path.
### board-meeting
- **Verdict: OPTIMIZE**
- Issues: (1) Phase 1 activation table + Phase 2 ordering cover 9 roles. (2) Invoked as `/cs:board` here but `/cs:boardroom` in c-level-agents — two names, one protocol. (3) Memory paths must match the canonical decision-memory layout chosen in domain finding 3.
- Verify: one command name used in both files (`grep -r "cs:board\b" c-level-advisor/` returns 0 or all-consistent); activation table lists 14 roles; memory paths match decision-logger.
### board-deck-builder
- **Verdict: OPTIMIZE**
- Issues: (1) `/board-deck [quarterly|monthly|fundraising]` doesn't exist as a command anywhere. (2) No CISO/GC/data sections cross-referenced to the 5 newer roles where relevant (minor). Content itself (4-act structure, bad-news framework, asks slide) is genuinely good.
- Verify: Quick Start either points at an existing command file or is rephrased as a trigger sentence; `references/deck-frameworks.md` + `templates/board-deck-template.md` exist (they do — keep green).
### competitive-intel
- **Verdict: OPTIMIZE**
- Issues: (1) Five phantom `/ci:*` commands as the entire Quick Start. (2) No script despite tabular scoring (threat matrix, feature-gap) being mechanizable — optional. Frameworks (8 tracking dimensions, win/loss interview protocol, over/under-tracking signals) are real.
- Verify: Quick Start contains no `/ci:` strings OR `commands/ci-*.md` files exist; battlecard template still referenced and present.
### cs-onboard
- **Verdict: OPTIMIZE**
- Issues: (1) Same output file as c-level-agents/onboard with a different schema (see domain finding 4). (2) `/cs:setup` + `/cs:update` names collide with the c-level-agents `/cs:` namespace without skill files behind them.
- Verify: exactly one interview skill owns `~/.claude/company-context.md`; context-engine "Required Context Fields" match the surviving schema; `grep -rl "company-context.md" c-level-advisor | xargs grep -l "project root"` → 0 hits.
### intl-expansion
- **Verdict: OPTIMIZE**
- Issues: (1) No script, no verification loop — ends at a checklist. (2) Market-selection scoring matrix is the mechanizable core; a 20-line stdlib scorer would lift this to KEEP. (3) Weakest A5 in the cross-cutting set: most rows ("research local buying behavior") a frontier model already knows.
- Verify: add `scripts/market_entry_scorer.py --sample` exits 0 emitting JSON with per-market weighted score; SKILL.md Quick Start invokes it; `references/regional-guide.md` retains region-specific regulatory facts (data residency, entity requirements) that aren't generic.
### ma-playbook
- **Verdict: OPTIMIZE**
- Issues: (1) Thinnest skill in domain (98 lines), pure checklist. (2) "2-15x ARR for SaaS" and "$1-3M per engineer" unsourced and freshness-fragile (A6). (3) No tool — a DD red-flag scanner or earnout-structure checker would be in-pattern with general-counsel-advisor. (4) Overlaps general-counsel-advisor (LOI/negotiation) and chief-data-officer-advisor (data diligence) without cross-references.
- Verify: multiples carry a source + as-of date; Adjacent Skills section cross-links gc-advisor + cdo-advisor; either a script exists passing `--help`, or the skill explicitly delegates quantitative work to cfo-advisor tools.
### executive-mentor/hard-call, postmortem, stress-test (3 skills)
- **Verdict: OPTIMIZE** (each)
- Issues: descriptions are literal placeholders (`"/em -hard-call — Framework for Decisions With No Good Options"`) — no trigger phrasing, malformed command name (`/em -hard-call` vs `/em:hard-call`). Bodies are excellent (10/10/10, Grove test, proper 5-Whys, change-register-with-verification-date); only the frontmatter fails.
- Verify: `python3 scripts/audit_skills.py` (repo validator) no longer flags these three for missing trigger; each description ≥ 1 "Use when" clause and < 1024 chars.
### c-level-agents (overview skill)
- **Verdict: OPTIMIZE**
- Issues: (1) Frontmatter `agents:` lists 8 (13 exist), `commands:` lists 17 (21 exist) — a new-gen router using this metadata will never surface vpe/cdo/caio/cco surfaces. (2) References block links `../../references/persona-voices.md` and `../references/persona-voices.md` inconsistently (one resolves, one doesn't).
- Verify: frontmatter agent/command lists match `ls agents/ | wc -l` = 13 and `ls skills/ | wc -l` − 1 = 21; all relative links resolve from the file's own directory.
### founder-mode
- **Verdict: OPTIMIZE**
- Issues: (1) Routing table has no signal rows for retention/CS (CCO), data architecture/training data (CDO), model selection/AI risk (CAIO), DORA/eng hiring (VPE) — the self-described "killer command" silently misroutes 4 domains to the wrong advisor. (2) Claims routing knowledge of decisions via decision-logger — path depends on domain finding 3.
- Verify: table includes the 4 missing roles with ≥ 4 keywords each; example "`the win rate dropped`" vs "`gross retention dropped`" route to CRO vs CCO respectively.
### onboard (c-level-agents)
- **Verdict: OPTIMIZE** — see domain finding 4. Verify: one canonical schema; symlink guidance (llm-wiki bridge) unchanged.
### brief
- **Verdict: OPTIMIZE**
- Issues: affected-roles checklist (drives boardroom panel composition) omits cs-general-counsel/cdo/caio/cco/vpe — a pricing-with-data-licensing brief can't seat the right panel.
- Verify: checklist lists all 14 advisors; `grep -c "cs-" skills/brief/SKILL.md` ≥ 14.
### decide
- **Verdict: OPTIMIZE**
- Issues: writes `~/.claude/decisions/{raw,approved}/` while decision-logger (the skill it claims to invoke) specifies `memory/board-meetings/`. The "two-layer memory" exists in two incompatible places.
- Verify: `grep -r "decisions/approved\|board-meetings/decisions" c-level-advisor/ -l` shows one canonical layout across decide, decision-logger, chief-of-staff, board-meeting.
### cdo-review, caio-review, cco-review, vpe-review (4 skills)
- **Verdict: OPTIMIZE** (each, same one-line fix)
- Issues: Routing sections send follow-ups to `/cs:chro-review` (and vpe-review also implies `/cs:coo-review`) — neither exists. Everything else is exemplary: role-specific forcing questions with thresholds, exact CLI to the backing skill's tools, structured output with verdict gates.
- Verify: every `/cs:*` string in the Routing sections resolves to a file under `c-level-agents/skills/*/SKILL.md`; `grep -r "cs:chro-review\|cs:coo-review" c-level-agents/` → 0 hits (or the two commands are created).
## KEEP-verdict verification criteria
- **ceo-advisor** — metrics dashboard targets present (burn multiple < 2x, NPS > 40); `python3 skills/ceo-advisor/scripts/strategy_analyzer.py` exits 0; boilerplate block replaced by agent-protocol pointer without losing the Tree-of-Thought section.
- **cfo-advisor** — all 3 scripts exit 0 bare; SKILL.md keeps burn-multiple/Rule-of-40/NDR thresholds table; "not a financial analyst skill" disambiguation to finance/ retained.
- **cto-advisor** — tech-debt priority formula `(Severity × Blast Radius) / Cost-to-fix` intact; both scripts exit 0; ADR template with 3-year-TCO checklist retained.
- **coo-advisor** — process-maturity 5-level table + both scripts green; VPE-vs-COO scope note added when role registry is fixed (advisory).
- **cpo-advisor** — D30 retention thresholds (20% consumer / 40% B2B) + invest/maintain/kill table intact; `pmf_scorer.py` exits 0.
- **cmo-advisor** — channel-level CAC discipline + pipeline-coverage 3–4x targets intact; both scripts green.
- **cro-advisor** — Magic Number + CAC-payback formulas verbatim; NRR benchmark table intact; both scripts green.
- **ciso-advisor** — `ALE = SLE × ARO` formula + compliance sequencing (SOC2 T1 → T2 → ISO/HIPAA) intact; both scripts green.
- **chro-advisor** — calibrated rating distribution table + compa-ratio 0.95–1.05 target intact; both scripts green.
- **chief-data-officer-advisor** (dual) — `ai_training_data_audit.py` bare-run exits 0 with GO/MITIGATE/NO-GO verdicts; both copies stay byte-identical (`diff -rq` clean); GDPR Art. 6 citations present in references.
- **chief-ai-officer-advisor** (dual) — 3 scripts exit 0; EU AI Act tier table retains Article citations; pricing figures carry an as-of date (A6 guard); copies identical.
- **chief-customer-officer-advisor** (dual) — `retention_decomposition_analyzer.py` flags leaky-bucket (NRR>100 ∧ GRR<85) on sample; GRR/NRR threshold table intact; copies identical.
- **general-counsel-advisor** (dual) — `contract_risk_scanner.py` bare-run flags ≥ 1 finding on bundled sample; "Not legal advice" disclaimer in SKILL.md + both tools; copies identical.
- **vpe-advisor** (dual) — `delivery_throughput_analyzer.py` emits DORA verdict + bottleneck (verified: exits 0, "Overall DORA level: High", bottleneck stage + % of cycle); fix workflow paths `../../skills/vpe-advisor/...` → `scripts/...`; copies identical.
- **context-engine** — 90-day staleness gate + anonymization never-send list intact; aligned to surviving onboarding schema.
- **decision-logger** — `python3 scripts/decision_tracker.py --demo` exits 0; DO_NOT_RESURFACE enforcement block intact; canonical path winner of domain finding 3.
- **scenario-war-room** — max-3-variables rule + cascade map + trigger-point examples intact; `scenario_modeler.py` exits 0; `/war-room` string removed or backed by a command.
- **org-health-diagnostic** — `health_scorer.py --json` emits machine-parseable dimension scores; 8-dimension thresholds + dimension-interaction table intact.
- **strategic-alignment** — `alignment_checker.py` detects orphans/conflicts/coverage-gaps on sample JSON; 5-people articulation test intact.
- **culture-architect** — values→behavioral-anchors table + culture-health score bands (80/65/50) intact.
- **company-os** — L10 agenda + IDS + rocks 3–7 cap intact; no phantom commands introduced.
- **founder-coach** — Skill×Will matrix + delegation ladder + calendar-audit target % table intact.
- **change-management** — ADKAR per-change-type timelines + resistance-pattern table intact.
- **internal-narrative** — audience translation matrix + contradiction check + 4-hour crisis rule intact.
- **executive-mentor / challenge / board-prep** — both scripts exit 0; challenge keeps assumption-confidence×impact matrix; board-prep keeps numbers-cold list.
- **office-hours / boardroom / execute / post-mortem / freeze / cross-eval** — pipeline artifact paths (`~/.claude/briefs|boardroom|execution|postmortems|freezes`) mutually consistent; each Routing section's `/cs:*` targets resolve; boardroom keeps Phase-2 isolation + dissent column; post-mortem keeps pre-committed-criteria scoring; freeze keeps default-period table.
- **cfo/cmo/cpo/cro/cto/ciso/gc-review** — every relative `python ../../../skills/...` path resolves from the file's directory; six questions per role remain role-specific (no copy-paste across roles); verdict gates (🟢/🟡/🔴) intact.
## Agents
| Agent | B1 (frontmatter) | B2 (differentiation) | B3 (body) | Top issue |
|---|---|---|---|---|
| cs-cfo-advisor | ⚠️ no "Use when" | PASS — burn-multiple/dilution forcing Qs, bear-case rule | PASS | model: opus justified? |
| cs-cmo-advisor | ⚠️ | PASS — one-sentence-positioning gate | **FAIL refs** — cites `growth_playbooks.md`, `marketing_operations.md` (don't exist; actual: growth_frameworks.md, marketing_org.md) | phantom KB files |
| cs-cro-advisor | ⚠️ | PASS — coverage>forecast, discount-creep tell | **FAIL refs** — all 3 KB names wrong (`revenue_operations/sales_motion/retention_expansion` vs actual sales_playbook/pricing_strategy/nrr_playbook) | phantom KB files |
| cs-cpo-advisor | ⚠️ | PASS — retention-curve-before-roadmap | **FAIL refs** — all 3 wrong (`product_vision/portfolio_strategy/pmf_framework` vs product_strategy/product_org_design/pmf_playbook) | phantom KB files |
| cs-coo-advisor | ⚠️ | PASS — DRI/cadence refusal gate | **FAIL refs** — all 3 wrong (`operating_cadence/okr_execution/scaling_playbooks` vs ops_cadence/process_frameworks/scaling_playbook) | phantom KB files |
| cs-chro-advisor | ⚠️ | PASS — no-promotion-without-ladder refusal | **FAIL refs** — all 3 wrong (`hiring_systems/comp_philosophy/leveling_ladders` vs people_strategy/comp_frameworks/org_design) | phantom KB files |
| cs-ciso-advisor | ⚠️ | PASS — assume-breach, $-quantified risk | **FAIL ref** — cites `threat_modeling.md` (doesn't exist; actual security_strategy.md) | phantom KB file |
| cs-chief-of-staff | ⚠️ | PASS — pure router, distinct job | **FAIL refs** — `routing_logic.md`/`synthesis_patterns.md` vs actual routing-matrix.md/synthesis-framework.md; routing table omits 5 roles | phantom KB files + role drift |
| cs-general-counsel-advisor | PASS (has disclaimer + scope) | PASS — escalate-to-counsel hard rule | PASS — exact tool wiring, correct paths | — |
| cs-cdo-advisor | PASS | PASS — "what decision does this data drive" refusal gate | PASS | — |
| cs-caio-advisor | PASS | PASS — no-eval-no-ship gate | PASS | — |
| cs-cco-advisor | PASS | PASS — gross-over-NRR, which-customer-would-you-fire | PASS | — |
| cs-vpe-advisor | PASS | PASS — explicit 4-way differentiation (CTO/eng-lead/CHRO/COO) | PASS | — |
| devils-advocate (executive-mentor) | no frontmatter at all (prose file) | PASS — exactly-3-concerns + never-clean-approval rules; behavior-changing | PASS — worked example | add YAML frontmatter |
**B2 assessment:** the personas genuinely pass the swap test. Each agent has different refusal gates (CFO: no scale on broken unit economics; CAIO: no eval set, no ship; CCO: no CS hire without a named customer outcome; CHRO: no promotion without ladder step), different tool wiring, and different success metrics — swapping cs-cfo-advisor's prompt into cs-cmo-advisor would be noticed immediately. The voice bookending ("opening/forcing/closing") is thin but the underlying workflows differ structurally.
**The systemic agent defect is B-side tool wiring, not differentiation:** 7 of 13 agents (the v2.5.0 batch: cfo is clean; cmo, cro, cpo, coo, chro, ciso, chief-of-staff are not) cite knowledge-base filenames that don't exist on disk. The 5 newer agents (gc, cdo, caio, cco, vpe — written against real files) are clean. Fix: one PR correcting ~16 filenames; verify with a link-checker pass (`for f in agents/*.md; do grep -o '\.\./\.\./skills/[a-z-]*/references/[a-z_-]*\.md' $f | while read p; do test -f "$(dirname $f)/$p" || echo "$f → $p"; done; done` → empty).
Also: `cs-ceo-advisor` and `cs-cto-advisor` live in `/agents/c-level/` outside this folder while 11 sibling links point at them via `../../../../agents/c-level/` — fragile but currently resolving.
## Plugin manifests
8 manifests, all schema-valid, all version 2.9.0 and consistent with marketplace.json (E1/E3 PASS). E2 findings:
| Plugin | Drift |
|---|---|
| c-level-skills (root) | Description says "33 skills + 13 agents + 21 commands" — accurate. But `"skills": ["./skills"]` only ships the bundle dir; the c-level-agents layer named in the description is NOT included in this plugin's skills path (it's a separate plugin) — description overpromises the install. |
| c-level-agents | Accurate (13 agents / 21 commands) — but the bundled overview SKILL.md frontmatter still says 8/17 (see per-skill). Manifest is ahead of its own skill. |
| executive-mentor | Accurate. CLAUDE.md claims it is "the only skill with a plugin.json (namespace: em)" — false since v2.5.x; 7 other plugin.json files exist in the domain. CLAUDE.md statement is the drift, not the manifest. |
| chief-ai-officer-advisor | Accurate, rich. "2026 pricing" claim is a freshness liability shared with the skill (A6). |
| chief-customer-officer-advisor / chief-data-officer-advisor / general-counsel-advisor / vpe-advisor | Accurate; descriptions correctly state "Standalone-installable; also bundled in c-level-skills" — the only place the dual-publication is documented. Mirror this sentence into the SKILL.md of each pair so editors learn about the twin copy. |
Scripts: all 25 bundle scripts pass repo-wide smoke tests (D1); spot-runs of `delivery_throughput_analyzer.py` (DORA verdict + bottleneck %), `decision_tracker.py --demo`, and `health_scorer.py --json` confirm deterministic, sample-embedded, machine-parseable output (D2/D3 PASS).
audit/newgen-2026-06/compliance.md
+219
View File
@@ -0,0 +1,219 @@
# Domain audit: ra-qm-team/ + compliance-os/ — new-gen model optimization
Audited: 2026-06-10 · Skills: 26 distinct (17 ra-qm-team incl. meta + 9 compliance-os; +2 verbatim sub-plugin duplicates) · Agents: 9 (8 compliance-os + cs-quality-regulatory) · Commands: 0 standalone (8 compliance-os skills are /cs:* command-shaped) · Plugins: 4 manifests (1 in marketplace)
## Scorecard
| Skill | Verdict | Top issue |
|---|---|---|
| ra-qm-team/capa-officer | OPTIMIZE | Cites 21 CFR 820.100 as current; removed by QMSR (eff. 2026-02-02) |
| ra-qm-team/eu-ai-act-specialist | OPTIMIZE | Embedded sample mis-teaches Art. 5(1)(f): tags RETAIL emotion recognition as prohibited (workplace/education only) |
| ra-qm-team/fda-consultant-specialist | REWRITE | Entire QSR section presents pre-QMSR 21 CFR 820 as current law; FY2024 user fees |
| ra-qm-team/gdpr-dsgvo-expert | OPTIMIZE | "30 days" deadlines (Art. 12(3) says one month + 2-month extension); WP29 framing; score-without-owner output |
| ra-qm-team/information-security-manager-iso27001 | OPTIMIZE | "Overall Compliance: 87%" auto-verdict example; thin clause citation; generic IR section |
| ra-qm-team/isms-audit-expert | KEEP | — |
| ra-qm-team/iso42001-specialist | KEEP | — (exemplary; template for the rest of the domain) |
| ra-qm-team/mdr-745-specialist | OPTIMIZE | PSUR table contradicts MDR Art. 86(1); IIb conformity-route table garbled; no Reg. 2023/607 transition note |
| ra-qm-team/qms-audit-expert | KEEP | — |
| ra-qm-team/quality-documentation-manager | OPTIMIZE | FDA table cites removed 820.40/.180/.181/.184/.186 as current |
| ra-qm-team/quality-manager-qmr | OPTIMIZE | Compliance matrix: "21 CFR 820 / QSR compliance" stale; "MPG/MPDG" half-stale |
| ra-qm-team/quality-manager-qms-iso13485 | OPTIMIZE | Record-retention table cites removed 820.30/.181/.184/.198 sections |
| ra-qm-team/ra-qm-skills (meta) | CUT-OR-MERGE | 66-line catalog with wrong paths and wrong skill count; adds no behavior |
| ra-qm-team/regulatory-affairs-head | OPTIMIZE | "~$22K (2024)" fee labeled current; QSR framing in pathway step 2 |
| ra-qm-team/risk-management-specialist | OPTIMIZE | ALARP w/ cost-benefit ("Proportionality") contradicts EU MDR "as far as possible" (EN ISO 14971:2019/A11:2021) |
| ra-qm-team/soc2-compliance | KEEP | — |
| ra-qm-team/compliance-team-eu-ai-act/* (dup) | CUT-OR-MERGE | Byte-identical copy of skills/eu-ai-act-specialist; drift risk |
| ra-qm-team/compliance-team-iso42001/* (dup) | CUT-OR-MERGE | Byte-identical copy of skills/iso42001-specialist; drift risk |
| compliance-os/compliance-os | KEEP | — (plugin.json description drift noted under Plugins) |
| compliance-os/compliance-readiness | KEEP | — |
| compliance-os/aims-audit | KEEP | — |
| compliance-os/ai-act-readiness | KEEP | — (phasing dates 2025-02-02/2025-08-02/2026-08-02/2027-08-02 correct) |
| compliance-os/gdpr-audit-prep | KEEP | — |
| compliance-os/iso27001-audit-prep | KEEP | — (typo: "Article 9.3" should be "Clause 9.3") |
| compliance-os/iso13485-audit-prep | KEEP | — |
| compliance-os/soc2-audit-prep | KEEP | — |
| compliance-os/fda-qsr-audit-prep | OPTIMIZE | Acknowledges QMSR yet cites removed 820.75/.100/.180/.198/.250 as live citations |
**Counts:** KEEP 13 · OPTIMIZE 11 · REWRITE 1 · CUT-OR-MERGE 1 (+2 duplicate copies)
## Domain-level findings
**1. The domain is two generations in one tree.** The 2026-05 wave (eu-ai-act-specialist, iso42001-specialist, all of compliance-os) is the best compliance work in the repo: Article/Clause-cited verdicts, explicit NOT-boundaries, deterministic tools with embedded samples, "Your Decision: [the call only the compliance officer can make]" output blocks, outside-counsel routing. The legacy 2025-era wave (the other 14 ra-qm-team skills) is competent practitioner content but pre-dates the citation-discipline pattern and has not been re-baselined against 2026 regulatory state.
**2. FDA QMSR is the single largest freshness failure (REWRITE/OPTIMIZE driver for 6 skills).** The FDA Quality Management System Regulation (final rule 89 FR 7496) replaced the QSR effective 2026-02-02 — four months before this audit. It incorporates ISO 13485:2016 by reference and REMOVES the subsection structure the legacy skills cite as current: 820.20/.30/.40/.50/.70/.75/.100/.180/.181/.184/.186/.198 no longer exist (retained/renumbered: 820.10 requirements, 820.35 records, 820.45 labeling+packaging; 21 CFR 801/803/806/830 unchanged). `fda-consultant-specialist` (skill + qsr_compliance_requirements.md reference + qsr_compliance_checker.py `--section 820.30` interface) has ZERO mentions of QMSR. Ironically, the compliance-os agent `cs-fda-qsr-auditor` and `fda-qsr-audit-prep` correctly state "substantially harmonized post-Feb 2026" — the fresh layer knows what the underlying skill it wraps does not.
**3. Citation precision: good-to-excellent in the new wave, mixed in legacy.** New-wave outputs cite Article+paragraph by contract ("do not paraphrase without cite"). Legacy skills cite at clause level (ISO 13485 4.2.3, 7.5.6, 8.5.2; GDPR Art. 6/9/35; MDR Annex II/VIII/XIV; MDCG 2019-11) — adequate — but carry concrete precision errors a notified-body auditor would flag: (a) mdr-745-specialist PSUR table says Class IIb "every 2 years" / IIa "when necessary" — MDR Art. 86(1) requires IIb+III at least annually and IIa at least every two years; (b) risk-management-specialist's ALARP framework includes "Cost-benefit of further reduction", which EU MDR Annex I §2 + the EN ISO 14971:2019/A11:2021 Z-annexes explicitly disallow (risk reduction "as far as possible" without economic consideration); (c) the eu-ai-act-specialist embedded sample tags a retail-store CCTV emotion-recognition system with `article_5_practice: emotion_recognition_in_workplace_or_education` — retail emotion recognition is Art. 50(3) transparency, not Art. 5 prohibited; the default `--sample` run teaches the wrong rule.
**4. Auto-decide vs route-to-human: NO skill auto-decides compliance verdicts; discipline is explicit in the new wave, implicit in legacy.** Every compliance-os and 2026-wave skill ends with a named-human handoff ("Your Decision: … compliance officer or legal counsel", "Outside Counsel Required" sections, cs-dpo-gdpr hard rule routing novel cases to GC). Legacy skills embed human signoff in workflows (CAPA approval signatures, "Classification confirmed with Notified Body", CER "reviewed by qualified evaluator") but lack an explicit handoff block, and two tool-output examples drift toward verdict-shaped numbers without an owner: information-security-manager's "Overall Compliance: 87%" and gdpr_compliance_checker's "Compliance score (0-100)". Not REWRITE-level — they are framed as prep/self-check tools — but every OPTIMIZE pass should add the new-wave "Your Decision" block.
**5. The "49 no-source references" repo flag is ~80% false positive in this domain.** 63 reference files in scope (55 distinct after sub-plugin dedup). Only ~8 have a formal "Sources" heading — which is what the repo validator keys on — but ~45 carry dense inline regulatory citations (Article/Clause/§/CFR/Annex markers; the AI Act and 13485 playbooks run 37–94 citation markers per file). Genuine zero-citation gaps ≈ 9 files, all generic-methodology docs: capa-officer/rca-methodologies.md + effectiveness-verification-guide.md, risk-management-specialist/risk-analysis-methods.md + risk-assessment-templates.md (77 lines, thin), quality-manager-qmr/quality-kpi-framework.md, information-security-manager-iso27001/incident-response.md, soc2-compliance/type1_vs_type2.md + evidence_collection_guide.md, qms-audit-expert/nonconformity-classification.md (1 marker). Fix: add a Sources block to those 9; do not bulk-rewrite the other 45.
**6. New-gen model lens.** Frontier models know ISO/GDPR/MDR basics; what earns context here is exactly what the new wave ships: clause-keyed gap analyzers with readiness verdicts, mock-audit scenario libraries (205 scenarios), evidence-reuse maps with confidence ratings, audit-prep interrogations with sample-driven "show me the record" questions. Legacy skills that mostly restate standard structure (info-sec manager's ISMS-implementation prose, parts of quality-manager-qmr) are the weakest A2 performers; their tools and checklists still earn their place.
**7. Cross-plugin coupling.** compliance-os skills invoke `../../../ra-qm-team/skills/*/scripts/*.py` by relative path. Works in the monorepo; breaks when plugins install independently (compliance-os plugin does not ship those scripts). Violates the repo "skills are self-contained" principle — at minimum document the ra-qm-skills co-install requirement in the manifest.
**8. Cruft.** 12 legacy `.zip` archives + `final-complete-skills-collection.md` committed at ra-qm-team/ root; ra-qm-team/CLAUDE.md says "14/14 skills" (folder has 16 + meta; omits eu-ai-act + iso42001 entirely, so the domain's own nav file hides its two best skills).
## Per-skill findings
### ra-qm-team/fda-consultant-specialist — REWRITE
Issues:
1. QSR section ("Quality System Regulation (21 CFR Part 820)") + subsystem table (820.20–820.181) presented as current; QMSR replaced this structure effective 2026-02-02 — wrong regulatory guidance in the highest-stakes lane.
2. `references/qsr_compliance_requirements.md` (753 lines) and `qsr_compliance_checker.py --section 820.30` interface built entirely on removed section numbers; zero QMSR mentions anywhere in the skill.
3. Fee table is FY2024 ($21,760 510(k) / $134,676 De Novo / $425,000+ PMA) with no fiscal-year label or MDUFA pointer.
4. Description still sells "QSR (21 CFR 820) compliance" — trigger text itself stale.
5. Pathway/eSTAR/cybersecurity/HIPAA content remains sound — structure salvageable, QSR third needs rebuild around ISO 13485-by-reference + retained 820.10/.35/.45 + unchanged 801/803/806/830.
Verify:
- `grep -ri QMSR ra-qm-team/skills/fda-consultant-specialist/ | wc -l` ≥ 5 (SKILL.md, description, qsr reference, checker help).
- `grep -rE '820\.(20|30|40|50|70|100|181|198)' SKILL.md references/` returns only lines explicitly marked historical/pre-2026.
- `python3 scripts/qsr_compliance_checker.py --help` exits 0 and help text names QMSR/ISO 13485, not "21 CFR 820 compliance" alone.
- Fee table rows carry an explicit FY label and "verify at fda.gov MDUFA" note.
### ra-qm-team/ra-qm-skills — CUT-OR-MERGE
Issues:
1. 66-line catalog page; duplicates README/plugin.json function; no workflow, no tools, no verification — fails A2/A4.
2. Says "12 skills" while the folder ships 16 and plugin.json says 14 — three conflicting counts.
3. Quick-start path `ra-qm-team/regulatory-affairs-head/SKILL.md` is wrong (missing `skills/` segment).
4. Omits eu-ai-act-specialist, iso42001-specialist, soc2-compliance from its table.
Verify:
- Folder removed (catalog content merged into ra-qm-team/README.md), OR rewritten as a real router; if kept: skill count matches `ls ra-qm-team/skills | wc -l` minus itself, and every path in the table resolves (`test -f` loop exits 0).
### ra-qm-team/risk-management-specialist — OPTIMIZE
Issues:
1. ALARP used as the acceptability framework incl. "Proportionality | Cost-benefit of further reduction" — EU MDR Annex I §2 + EN ISO 14971:2019/A11:2021 Z-annexes prohibit economic considerations; for CE-marked devices the criterion is "as far as possible" (AFAP). A notified body flags this exact table.
2. ISO 14971:2019 itself dropped ALARP from the normative body; skill presents it as the standard's method.
3. `references/risk-analysis-methods.md` + `risk-assessment-templates.md` (77 lines) cite zero sources.
4. No explicit named-human handoff for residual-risk acceptance (it's implied via "management signoff" only in iso42001-specialist, not here).
Verify:
- `grep -c 'as far as possible\|AFAP' SKILL.md` ≥ 2 and ALARP appears only with an explicit "non-EU / not acceptable under MDR" caveat.
- `grep -c 'Cost-benefit' SKILL.md` = 0 in the EU acceptability context.
- `python3 scripts/risk_matrix_calculator.py -p 4 -s 5 --output json` exits 0, emits `risk_level` key.
### ra-qm-team/mdr-745-specialist — OPTIMIZE
Issues:
1. PSUR table wrong vs MDR Art. 86(1): says IIb "Every 2 years", IIa "When necessary" — regulation requires IIb (all, not just implantable) at least annually, IIa at least every 2 years.
2. Conformity-route row "IIb | Annex IX + X or X + XI" garbled (routes are Annex IX, or Annex X+XI).
3. No mention of Reg. (EU) 2023/607 extended transition (legacy MDD devices to 2027/2028) — the question every MDR client asks first in 2026.
4. PMS table cites "PMS Plan | Article 84" correctly but omits Art. 83 (system) and Art. 86 (PSUR) cites where the schedule lives.
Verify:
- PSUR table matches Art. 86(1) verbatim cadence (`grep -A4 'PSUR Schedule' SKILL.md` shows IIb=annual, IIa=every 2 years).
- `grep -c '2023/607' SKILL.md references/` ≥ 1.
- `python3 scripts/mdr_gap_analyzer.py --device Test --class IIa --output json` exits 0 with gap list.
### ra-qm-team/eu-ai-act-specialist — OPTIMIZE
Issues:
1. Embedded sample system "Emotion recognition in retail store CCTV" is hard-tagged `article_5_practice: emotion_recognition_in_workplace_or_education` → default `--sample` output declares retail emotion recognition PROHIBITED. Correct treatment: Art. 50(3) transparency (limited-risk). Wrong teaching in the default demo of a flagship skill.
2. Classifier trusts caller-supplied `article_5_practice` flags rather than deriving from context fields it already collects (`users`, `intended_purpose`) — at minimum the docstring should state the flag is the user's legal pre-determination.
3. Verbatim duplicate at compliance-team-eu-ai-act/ (see Plugins).
Verify:
- `python3 scripts/ai_system_risk_classifier.py` sample output classifies the retail-CCTV system as LIMITED-RISK citing Art. 50(3), or the sample is changed to a genuine workplace context.
- All 3 scripts exit 0 on `--help` and bare run; every verdict line contains "Article".
### ra-qm-team/gdpr-dsgvo-expert — OPTIMIZE
Issues:
1. Rights table + body say "30 days" / "extendable to 90" — Art. 12(3) is one month, extendable by two further months; in a deadline-tracking tool the month/30-day distinction loses up to 3 days.
2. "WP29 high-risk criteria" — EDPB-endorsed but should be cited as EDPB/WP248 rev.01.
3. Compliance checker emits 0-100 score with no named-DPO routing block; SKILL.md has no "Your Decision" handoff (contrast gdpr-audit-prep which does this right).
4. No mention of EU-US Data Privacy Framework / Chapter V transfer tooling in SKILL.md (playbook reference covers it; surface a pointer).
Verify:
- `grep -c 'one month' SKILL.md` ≥ 1; `grep -c '30 days' SKILL.md` = 0 in the Art. 12 deadline context.
- `python3 scripts/data_subject_rights_tracker.py add --type access --subject T --email t@x.de` then `list` exits 0; due-date computed by calendar month.
- SKILL.md gains an output block routing final determinations to DPO/counsel.
### ra-qm-team/information-security-manager-iso27001 — OPTIMIZE
Issues:
1. Worked example ends "Overall Compliance: 87%" with no owner/handoff — the closest thing to an auto-verdict in the domain.
2. Body is clause-thin (only 6.1.2 cited); 2022 control IDs appear only in the example; A5 weak for a new-gen model (ISMS-implementation prose a frontier model already knows).
3. `references/incident-response.md` (420 lines) cites zero sources and duplicates engineering-team incident-response ground.
4. CLI surface in SKILL.md (`--template healthcare`, `--domains`) must be verified against actual argparse (legacy doc drift risk).
Verify:
- Every documented flag exists: `python3 scripts/risk_assessment.py --help` and `compliance_checker.py --help` list `--scope/--template/--standard/--gap-analysis`.
- Worked example ends with a named-human review step (ISMS owner / CISO) instead of bare percentage.
- incident-response.md gains a Sources block (≥3: ISO 27035, NIST SP 800-61r3, A.5.24-26) or is cut in favor of a pointer.
### ra-qm-team/capa-officer — OPTIMIZE
Issues:
1. "FDA 21 CFR 820.100" requirements section presents removed regulation as current (QMSR: CAPA now flows through ISO 13485 8.5.2/8.5.3 incorporated by reference).
2. `references/rca-methodologies.md` + `effectiveness-verification-guide.md` (917 lines combined) cite zero sources.
3. Otherwise the strongest legacy skill (decision trees, validated 5-Why example, metrics with formulas) — targeted edits only.
Verify:
- 820.100 section reframed as "pre-2026 QSR / now via ISO 13485 8.5 under QMSR" (`grep -c QMSR SKILL.md` ≥ 1).
- `python3 scripts/capa_tracker.py --sample > /tmp/c.json && python3 scripts/capa_tracker.py --capas /tmp/c.json --output json` exits 0 with summary metrics keys.
### ra-qm-team/quality-documentation-manager — OPTIMIZE
Issues:
1. "FDA 21 CFR 820" table (820.40/.180/.181/.184/.186) presents removed sections as current; under QMSR records requirements live in 820.35 + ISO 13485 4.2.4/4.2.5.
2. Part 11 content is solid and unaffected — single-table fix plus reference sweep of 21cfr11-compliance-guide.md for cross-refs into old 820.
Verify:
- FDA table updated to QMSR structure; `grep -E '820\.(40|181|184|186)' SKILL.md` only in historical context.
- `python3 scripts/document_validator.py --sample > /tmp/d.json && python3 scripts/document_validator.py --doc /tmp/d.json --output json` exits 0.
### ra-qm-team/quality-manager-qms-iso13485 — OPTIMIZE
Issues:
1. Record-retention table regulatory basis column cites removed 820.181/.184/.30/.198.
2. Otherwise strong (exclusion table, validation standards ISO 11135/11137/17665, supplier scoring) — single-table fix.
Verify:
- Retention table bases updated to QMSR/ISO 13485 cites.
- `python3 scripts/qms_audit_checklist.py --clause 7.3` exits 0 and emits 7.3-specific questions.
### ra-qm-team/quality-manager-qmr — OPTIMIZE
Issues:
1. Multi-jurisdiction matrix: "USA | 21 CFR 820 | FDA registration, QSR compliance" stale post-QMSR; "Germany | MPG/MPDG" — MPG repealed 2021, list MPDG/MPEUAnpG only.
2. Generic culture-survey and KPI content is the domain's weakest A2 (frontier model knows it); KPI reference cites zero sources.
Verify:
- Matrix row reads QMSR; `grep -c 'MPG/' SKILL.md` = 0.
- `python3 scripts/management_review_tracker.py --help` exits 0.
### ra-qm-team/regulatory-affairs-head — OPTIMIZE
Issues:
1. Pathway matrix fees "~$22K (2024)" — stale FY presented as current; needs FY label + MDUFA pointer.
2. Step 2 lists "FDA (US): 21 CFR Part 820" as applicable regulation without QMSR framing.
3. Overlap with mdr-745-specialist + fda-consultant-specialist is acceptable (strategy vs execution split) but should cross-link rather than restate the SE table.
Verify:
- Fee cells carry FY labels; `grep -c QMSR SKILL.md` ≥ 1.
- `python3 scripts/regulatory_tracker.py --help` exits 0.
### compliance-os/fda-qsr-audit-prep — OPTIMIZE
Issues:
1. Correctly states post-Feb-2026 harmonization, then cites removed sections as live law throughout (820.198, 820.75, 820.100, 820.180, 820.250) — internally inconsistent; the right cites are ISO 13485 clauses (8.2.2, 7.5.6, 8.5.2, 4.2.5) + retained 820.35/.45 + unchanged 803/801/830/806.
2. Workflow shells into fda-consultant-specialist scripts whose interfaces are themselves pre-QMSR (`--section 820.30`) — blocked on the REWRITE above.
Verify:
- Each of the six questions cites the QMSR-era source (ISO 13485 clause or retained CFR section); `grep -E '820\.(75|100|180|198|250)' SKILL.md` only with "pre-QMSR" annotation.
- Workflow paths resolve after the fda-consultant-specialist rewrite.
### Duplicate sub-plugins (compliance-team-eu-ai-act/, compliance-team-iso42001/) — CUT-OR-MERGE
Issues:
1. `diff -r` confirms byte-identical copies of ra-qm-team/skills/{eu-ai-act,iso42001}-specialist — two sources of truth; the Art. 5 sample bug must now be fixed twice.
2. Neither sub-plugin is registered in marketplace.json, so the duplication currently buys nothing.
Verify:
- Either sub-plugins deleted (standalone install served by marketplace entry pointing at the skills/ copy), or a sync script/CI check asserts `diff -r` emptiness on every PR.
## KEEP-verdict verification criteria
- **iso42001-specialist:** `python3 scripts/aims_gap_analyzer.py` exits 0, prints `Certification readiness:` ∈ {ready, stage_2_candidate, not_ready} + weighted coverage %; all 3 tools pass bare run; every gap line carries a Clause number.
- **isms-audit-expert:** `python3 scripts/isms_audit_scheduler.py --year 2026 --format markdown` exits 0 and emits a quarter-bucketed plan; finding template retains Requirement/Evidence/Gap triple.
- **qms-audit-expert:** `python3 scripts/audit_schedule_optimizer.py --interactive` help path exits 0; clause-scope table keeps 4.2→8.5 coverage; classification decision tree intact.
- **soc2-compliance:** `python3 scripts/control_matrix_builder.py --categories security --format json` exits 0 with ≥1 control per CC1–CC9; gap_analyzer distinguishes type1/type2 modes.
- **compliance-os (orchestrator):** all 4 tools exit 0 on bare run; `audit_simulator.py` output reports `healthy=` against the ≥40% observation / ≤15% critical rule; `assets/mock_audit_library.json` parses with 205 scenarios.
- **compliance-readiness:** output template retains the 🟢/🟡/🔴 verdict + "Top 3 Actions with owners" + routing block; all 4 workflow script paths resolve from the skill dir.
- **aims-audit:** 6 questions each name a Clause or Annex A control; workflow paths into ra-qm-team resolve; routes verdict to `/cs:decide`.
- **ai-act-readiness:** phasing dates remain exactly 2025-02-02 / 2025-08-02 / 2026-08-02 / 2027-08-02; "Legal Review Required" section retained; every question keeps its Article cite.
- **gdpr-audit-prep:** Art. 12(3) one-month language retained; Art. 30/35(7)/33(5) cites intact; "Outside Counsel Required" section retained.
- **iso27001-audit-prep:** fix "Article 9.3"→"Clause 9.3"; 3-year-coverage question + auditor-independence check retained; scheduler path resolves.
- **iso13485-audit-prep:** Clause cites (8.2.4, 7.5.6, 5.6.2/5.6.3) intact; DHF-sampling question retains stratification by class.
- **soc2-audit-prep:** observation-period discipline questions (cycle skips, first-month evidence, exception materiality) retained; AT-C 205 cite intact.
## Agents
All 9 pass B1–B3. The 8 compliance-os personas are the best-differentiated agent set audited: each has a distinct voice that changes behavior (cs-ciso-iso27001 "samples, not demos"; cs-dpo-gdpr refuses to paraphrase the Regulation; cs-fda-qsr-auditor tracks Form 483 gradient distinct from ISO NC grades), explicit vs-sibling differentiation paragraphs, and hard rules that route novel/legal calls to GC or outside counsel — the route-to-human discipline the legacy skills lack lives here. cs-fda-qsr-auditor is also the only artifact in either domain that correctly states the QMSR transition.
Issues: (1) cs-fda-qsr-auditor still cites removed 820.x section numbers in its forcing questions — inherits the fda-qsr-audit-prep fix. (2) All 8 pin `model: opus` — pre-Fable-era pin; revisit per repo model policy. (3) `skills:` frontmatter uses repo-relative paths (`ra-qm-team/skills/...`) that break outside the monorepo. (4) cs-quality-regulatory (agents/ra-qm-team/) is generic by comparison (sonnet, catalog-style body, no voice/forcing questions, no QMSR awareness) — OPTIMIZE to the compliance-os persona pattern, and its skill paths omit the `skills/` segment.
## Plugin manifests
| Manifest | E1 schema | E2 description | E3 marketplace |
|---|---|---|---|
| ra-qm-team/.claude-plugin/plugin.json | PASS (`"skills": ["./skills"]`) | DRIFT — "14 skills"; folder ships 16 + meta (eu-ai-act + iso42001 + soc2 uncounted) | Registered (v2.9.0) |
| compliance-os/.claude-plugin/plugin.json | PASS (9 explicit paths) | DRIFT — claims "9 supported frameworks" + "3 cs-* agents + 3 commands"; SKILL.md/tools support 12 frameworks, 8 agents, 8 command-skills ship | **NOT in marketplace.json** (66 plugins, none sourced from ./compliance-os) |
| ra-qm-team/compliance-team-eu-ai-act/plugin.json | PASS | OK | **NOT in marketplace.json** |
| ra-qm-team/compliance-team-iso42001/plugin.json | PASS | OK | **NOT in marketplace.json** |
Findings: (1) Three of four manifests are orphans — built as plugins, never registered; either register them or delete the sub-plugin duplicates and fold compliance-os registration into the next marketplace bump. (2) compliance-os skills depend at runtime on ra-qm-team scripts via `../../../` paths — manifest must declare the co-install requirement or vendor the scripts. (3) ra-qm-team ships 12 stale `.zip` skill archives + `final-complete-skills-collection.md` at domain root — remove from the public tree. (4) ra-qm-team/CLAUDE.md ("14/14 skills", omits the domain's two flagship 2026 skills) and root CLAUDE.md ("18 RA/QM skills") disagree with each other and with the folder; reconcile counts in one pass.
audit/newgen-2026-06/cross-cutting.md
+179
View File
@@ -0,0 +1,179 @@
# Cross-cutting audit: root agents, commands, standards, templates, marketplace, CI — new-gen model optimization
Audited: 2026-06-10
Scope: root `agents/` (32), root `commands/` (39), `standards/`, `templates/`, `orchestration/`, `custom-gpt/`, `assets/`, `.claude-plugin/marketplace.json`, `scripts/`, `.github/workflows/`, root README.md + CLAUDE.md. Rubric dimensions B/C/D/E.
---
## Counter-drift reconciliation table
Measured today (canonical tree, excluding `.codex/.gemini/.hermes/.vibe` sync copies and `docs/`):
| Metric | **Actual (measured)** | README.md | CLAUDE.md header (L9) | CLAUDE.md v2.10.3 block | CLAUDE.md footer (L515) | marketplace.json metadata | agents/CLAUDE.md |
|---|---|---|---|---|---|---|---|
| Skills (SKILL.md) | **346** (`audit_skills.py` count) / 347 (`find`) | 338 | 338 | 343 | 338 | 343 | "42 production skills" AND "177 existing skills" (same page) |
| Marketplace plugins | **66 entries** in marketplace.json; **77 plugin.json manifests** on disk (78 incl. `.codex-plugin` sync artifact) | — | 62 | 64 | 62 | **claims 64 — its own `plugins` array has 66** | — |
| Python tools (in-skill `scripts/*.py`) | **555** | 533 | 533 | 548 | — | 548 | — |
| Reference docs (`references/*.md`) | **700** | 676 | 676 | 691 | — | 691 | — |
| Agents (repo-wide / root) | **92 / 32** | 51+ | 51+ ("32 standalone") | — | — | 51+ | "16 Agents Currently Available" (table actually lists 19; folder has 32) |
| Slash commands (repo-wide / root) | **99 / 39** | 87+ | 87+ | 90+ | — | 90+ | — |
| Domains | **17** top-level skill domains | 16 | 16 | 17 | 16 | 17 | — |
| Version | — | — | — | v2.10.3 | **v2.9.0** | v2.10.3 | — |
Key drift facts:
1. **marketplace.json is internally inconsistent**: `metadata.description` and `description` both say "64 marketplace plugins" while the `plugins` array contains **66** entries (v2.10.x additions `universal-scraping-architect` and `youtube-full` were appended without bumping the counter).
2. **11 plugin.json manifests exist on disk but are NOT registered in marketplace.json** (invisible to marketplace installs):
`compliance-os/`, `engineering-team/snowflake-development/`, `engineering/behuman/`, `engineering/claude-coach/`, `engineering/grill-with-docs/`, `engineering/llm-cost-optimizer/`, `engineering/prompt-governance/`, `finance/business-investment-advisor/`, `marketing-skill/video-content-strategist/`, `ra-qm-team/compliance-team-eu-ai-act/`, `ra-qm-team/compliance-team-iso42001/`. (Plus `.codex-plugin/plugin.json`, a sync artifact.) Notably, `compliance-os` is *named as a domain* in the marketplace description yet has no marketplace entry.
3. **CLAUDE.md disagrees with itself in three places** (header 338/62/16, v2.10.3 block 343/64/17, footer v2.9.0/338/62/16). README still carries the pre-v2.10 numbers everywhere, including the shields badge (`Skills-338`).
4. Even the freshest claimed numbers (343/64/548/691) trail reality (346/66-77/555/700) — the counters were last trued up at v2.10.3 and have drifted again.
---
## Root agents
32 agents (25 `cs-*` + 7 personas). **17 of 32 (53%) lack trigger phrasing** ("Use when…" / "Spawn when…" / "Invoke via…") in `description` — B1 fail. All but 2 agents carry the identical generic `tools: [Read, Write, Bash, Grep, Glob]` list (B1 "minimal tools" not practiced). The personas use a non-standard frontmatter schema (`name` with spaces, `color`, `emoji`, `vibe` fields — not Claude Code agent fields).
| Agent | Verdict | Issue |
|---|---|---|
| engineering/cs-backend-engineer | KEEP | Exemplary: trigger + invocation path + `context: fork` + differentiated forcing questions |
| engineering/cs-frontend-engineer | KEEP | Same pattern; good |
| engineering/cs-fullstack-engineer | KEEP | Same pattern; good |
| engineering/cs-karpathy-reviewer | KEEP | Good trigger; near-duplicate of `engineering/karpathy-coder/agents/karpathy-reviewer.md` (differs, but two sources of truth) |
| engineering/cs-wiki-ingestor / -librarian / -linter (3) | KEEP | Good triggers; near-duplicates of `engineering/llm-wiki/agents/wiki-*.md` — pick one canonical home |
| engineering/cs-senior-engineer | OPTIMIZE | Trigger OK but scope ("architecture, code review, DevOps, API design") overlaps cs-engineering-lead + the 3 role engineers — B2 differentiation weak |
| engineering-team/cs-engineering-lead | OPTIMIZE | Trigger OK; differentiate from cs-senior-engineer or merge |
| engineering-team/cs-workspace-admin | KEEP | Specific, trigger present |
| marketing/cs-aeo | KEEP | Model trigger + voice + refusal rule |
| marketing/cs-webinar-marketer | KEEP | Model trigger + voice |
| marketing/cs-content-creator | OPTIMIZE | No trigger phrasing; description is a topic list |
| marketing/cs-demand-gen-specialist | OPTIMIZE | No trigger phrasing |
| c-level/cs-ceo-advisor | OPTIMIZE | No trigger; 360-line body is content-rich but description is a noun phrase |
| c-level/cs-cto-advisor | OPTIMIZE | No trigger |
| product/cs-product-manager | OPTIMIZE | No trigger; claims 8 skills incl. those owned by sibling agents (strategist, ux-researcher) — B2 overlap |
| product/cs-product-strategist | OPTIMIZE | No trigger; skill overlap with cs-product-manager |
| product/cs-agile-product-owner | OPTIMIZE | No trigger |
| product/cs-ux-researcher | OPTIMIZE | No trigger |
| product/cs-product-analyst | REWRITE | No trigger AND 31-line near-placeholder body (B3) — thinnest agent in the folder |
| project-management/cs-project-manager | OPTIMIZE | No trigger; 515-line body |
| business-growth/cs-growth-strategist | KEEP | "Spawn when…" present |
| finance/cs-financial-analyst | KEEP | "Spawn when…" present |
| ra-qm-team/cs-quality-regulatory | KEEP | "Spawn when…" present |
| personas/solo-founder | OPTIMIZE | No trigger; no `skills:` mapping; non-standard frontmatter |
| personas/startup-cto | OPTIMIZE | No trigger; overlaps cs-cto-advisor (B2 pair) |
| personas/growth-marketer | OPTIMIZE | No trigger; no `skills:`; overlaps cs-demand-gen-specialist |
| personas/content-strategist | OPTIMIZE | No trigger; overlaps cs-content-creator |
| personas/product-manager | OPTIMIZE | No trigger; overlaps cs-product-manager |
| personas/finance-lead | OPTIMIZE | No trigger; skills list (`ceo-advisor`, `cost-estimator`) is bare-name shorthand that doesn't resolve to paths; overlaps cs-financial-analyst |
| personas/devops-engineer | OPTIMIZE | No trigger; references `ms365-tenant-manager` / `aws-solution-architect` which exist only as **.zip archives** in engineering-team/ |
Cross-cutting agent issues:
- **`skills:` frontmatter shorthand is one directory level stale repo-wide**: e.g. `product-team/product-manager-toolkit` — actual path is `product-team/skills/product-manager-toolkit`. Every domain-prefixed `skills:` value resolves only via the `skills/` insertion.
- `templates/agent-template.md` is the root cause of the missing-trigger pattern: it mandates "One-line description… under 150 characters" with **no trigger-phrase requirement**, contradicting agents/CLAUDE.md's own current guidance and rubric B1.
---
## Root commands
39 commands. **34 of 39 never reference `$ARGUMENTS`** despite advertising `Usage: /cmd <args>` in descriptions (C2 weak; only focused-fix, tc, and the 4 cs-*-review/grill commands handle it). Only the 4 newest commands use `argument-hint`.
**Systemic C3 defect — phantom script paths: 28 of 39 commands contain ≥1 path that does not exist.** Root commands consistently reference `<domain>/<skill>/scripts/x.py`, but skills were reorganized into `<domain>/skills/<skill>/scripts/x.py` (and single-skill plugins into `<plugin>/skills/<name>/scripts/`). Every "Scripts" section pointing at e.g. `engineering/changelog-generator/...`, `finance/financial-analyst/...`, `product-team/product-manager-toolkit/...` is a dead path; the scripts exist one level deeper. A model following these commands hits file-not-found on first invocation.
| Command | Verdict | Issue |
|---|---|---|
| plugin-audit | KEEP | Real 8-phase orchestration; 4 stale paths to fix |
| seo-auditor | KEEP | Real pipeline; 8 stale paths |
| tc | KEEP | State machine + $ARGUMENTS dispatch; 6 stale paths |
| cs-aeo, cs-webinar | KEEP | Tool-wired workflows |
| cs-backend-review, cs-frontend-review, cs-fullstack-review, cs-engineer-grill | KEEP | argument-hint + agent fork + gates — the model pattern for the rest |
| a11y-audit | KEEP | Uses `{skill_path}` placeholder (resilient); fix Skill Reference path |
| code-to-prd | KEEP | `{skill_path}` placeholder pattern; 3 stale refs |
| chaos-experiment, slo-design, operator-audit, flag-cleanup | KEEP | Interactive wizards/gates; no stale paths detected |
| google-workspace | OPTIMIZE | Good content; 6 stale script paths |
| changelog, pipeline, okr, rice, persona, retro, user-story, saas-health, tech-debt, competitive-matrix, financial-health, project-health, sprint-health | OPTIMIZE | Script-wired (passes C3 in spirit) but **all script paths are phantoms** (missing `skills/` segment) |
| focused-fix | OPTIMIZE | Strong 5-phase protocol + $ARGUMENTS; "Related Skills" points to `engineering/focused-fix` (actual: `engineering/skills/focused-fix`) and to external `superpowers:systematic-debugging` (not in this repo). **Known-issue verified FALSE POSITIVE: the "TODO/FIXME" string is instructional content in the Phase 3 diagnostic checklist, not a real marker** — no actual TODO/FIXME/placeholder markers in commands/ or agents/ |
| karpathy-check | CUT-OR-MERGE (dedupe) | **Byte-identical** to `engineering/karpathy-coder/commands/karpathy-check.md`; also references bare `scripts/complexity_checker.py` which only resolves inside the plugin |
| wiki-ingest, wiki-init, wiki-lint, wiki-query, wiki-log (5) | CUT-OR-MERGE (dedupe) | **Byte-identical** to `engineering/llm-wiki/commands/wiki-*.md` — root copies are second sources of truth that will drift |
| prd | CUT-OR-MERGE | Bare-prompt restatement: 25 lines = output bullet list + skill pointer. A frontier model produces this without the command |
| sprint-plan | CUT-OR-MERGE | Same: 25 lines, no script, no gate, no $ARGUMENTS |
| tdd | CUT-OR-MERGE | References 4 scripts explicitly labeled "(library module)" — not CLI-invocable, so the command orchestrates nothing a bare prompt couldn't |
**Cut-or-merge candidates: 9** (6 byte-identical duplicates + prd + sprint-plan + tdd).
---
## standards/ + templates/ + orchestration/ + custom-gpt/ + assets/
**standards/** — NOT orphaned: referenced by `.github/workflows/skill-quality-review.yml`, `skill-security-audit.yml`, `commands/plugin-audit.md`, root CLAUDE.md, `.claude/commands/update-docs.md`. No stale model names or dead dates. Caveat: `communication-standards.md` is only **38 lines** (vs 319–545 for siblings) — underweight for a "standards" file. The other 4 are substantive.
**templates/** — Stale meta-documentation:
- `templates/CLAUDE.md` says `agent-template.md` exists "**(when created)**" — it has existed for a long time; also promises `command-template.md` and workflow templates "(when created)" that were **never created** (phantom inventory).
- `templates/agent-template.md` (318 lines) actively teaches the B1 anti-pattern: "description… under 150 characters", no trigger-phrase requirement, generic 5-tool list. This template is why 17/32 root agents fail B1. Verdict: REWRITE the frontmatter section.
**orchestration/ORCHESTRATION.md** (262 lines) — Referenced from README + agents/CLAUDE.md + mkdocs (not orphaned). Concept is sound, but its load examples are stale: `engineering/aws-solution-architect/SKILL.md` and `engineering/mcp-server-builder/SKILL.md` — `aws-solution-architect` exists only as `engineering-team/aws-solution-architect.zip` (an archive!) and mcp-server-builder lives at `engineering/skills/mcp-server-builder/`. Verdict: OPTIMIZE (fix example paths).
**custom-gpt/README.md** (128 lines) — Referenced from README + mkdocs; 6 ChatGPT GPT links. Not orphaned; external links unverifiable from here. Verdict: KEEP.
**assets/icon.png** — **Orphan.** Zero references from marketplace.json, any plugin.json, README, or mkdocs (grep across .json/.md/.yml). Either wire it into marketplace metadata or remove.
**engineering-team/*.zip** (12 archives: senior-fullstack.zip, aws-solution-architect.zip, etc.) — observed while tracing paths; zipped skill archives sitting in the published tree, referenced by personas/orchestration as if unzipped. Flagging for the engineering-team domain auditor.
---
## scripts/ (build & sync)
| Script | --help | Finding |
|---|---|---|
| check_plugin_json.py | PASS (argparse) | `--all` validates all 77 manifests, 0 WARN/FAIL today. Solid (D1–D3) |
| sync-codex-skills.py | PASS | Knows all 17 domains incl. markdown-html |
| sync-gemini-skills.py | PASS | **Missing `markdown-html` domain** — 5 skills never sync to Gemini |
| sync-hermes-skills.py | PASS | **Missing `markdown-html`** — never syncs to Hermes |
| sync-vibe-skills.py | PASS | **Missing `markdown-html`** — never syncs to Vibe |
| sync_skill_bundles.py | PASS (argparse) | OK |
| extract_release_notes.py | PASS (argparse) | OK |
| **audit_skills.py** | **FAIL** | No argparse; `--help` is ignored and the **full 346-skill audit runs** (~30s). D1 violation in the repo's own meta-audit tool |
| **generate-docs.py** | **FAIL — destructive** | No argparse; `--help` silently **regenerates the entire docs/ tree** (verified: modified 5 + created 4 files during this audit; reverted). A help invocation must never write. Also lacks markdown-html coverage (no docs pages exist for the 5 markdown-html skills) |
| convert.sh / install.sh / openclaw-install.sh / review-new-skills.sh / *-install.sh | header-documented | No hardcoded `/Users/...` or maintainer-specific paths found anywhere in scripts/ (good). `convert.sh` also has no markdown-html awareness |
---
## CI workflows
12 workflows. What actually gates PRs:
- **ci-quality-gate.yml** — runs on every PR. **CONFIRMED: `python scripts/check_plugin_json.py --all` runs as a blocking step**, exactly as CLAUDE.md claims (labeled "guards #539 + #686"). yamllint + workflow schema check (schema check is `|| true` — advisory). **Gap: the blocking `compileall` step covers only 9 pre-v2.7 folders** (`marketing-skill product-team c-level-advisor engineering-team ra-qm-team engineering business-growth finance project-management scripts`) — **8 newer domains are never syntax-checked in CI**: `productivity/`, `marketing/`, `research/`, `research-ops/`, `business-operations/`, `commercial/`, `markdown-html/`, `compliance-os/`. Safety scan and link check are `|| true` (advisory).
- **skill-quality-review.yml / skill-security-audit.yml** — PR-triggered, reference standards/; Tessl review depends on external secrets.
- **enforce-pr-target.yml** — pull_request_target gate (branch strategy enforcement). Runs.
- **claude.yml / claude-code-review.yml / pr-issue-auto-close.yml / virustotal-scan.yml** — event-driven; not quality gates per se.
- **release.yml** (push→main), **static.yml** (Pages), **sync-codex-skills.yml** (push), **smart-sync.yml** (issues; excluded from schema validation due to `projects_v2_item`).
Net: the only **blocking** code-quality gates are compileall (with the 8-domain hole) and check_plugin_json. Everything else is advisory or external-dependent. Note: `tests/` pytest suite is maintainer-local (gitignored) — no test gate runs in CI by design, but CLAUDE.md's claim "run locally; not in CI" is accurate.
---
## README/CLAUDE.md accuracy
- README header, badge, and FAQ all say **338 skills / 533 tools / 676 references / 51+ agents / 87+ commands / 16 domains** — all five numbers are stale (actual: 346 / 555 / 700 / 92 / 99 / 17). The shields badge hardcodes `Skills-338`.
- README relative links: all resolve in-tree (CHANGELOG.md, CONTRIBUTING.md, personas, orchestration). **No 404-for-cloners links found beyond the documented gitignored-folder exception.** CLAUDE.md's `documentation/...` links are covered by the Maintainer-Local Folders note (intentional).
- CLAUDE.md is **self-contradictory**: header (L9) 338/62/16, v2.10.3 block 343/64/17, footer "Last Updated May 27, 2026 / Version v2.9.0 / 338 skills…16 domains / 62 plugins". Three different repo states in one file.
- agents/CLAUDE.md is the worst offender: "42 production skills" and "177 existing skills" in adjacent paragraphs, "16 Agents Currently Available" heading over a 19-row table, while the folder holds 32 agent files.
- README claim "All 533 Python CLI tools… verified to run with `--help`" — falsified twice in scripts/ alone (audit_skills.py, generate-docs.py), and the count is 555.
---
## Top recommendations
1. **Single-source the counters.** Add a `scripts/update_counters.py` (or extend audit_skills.py) that computes skills/plugins/tools/references/agents/commands from the tree and rewrites README badge, CLAUDE.md header+footer, and marketplace.json metadata in one pass; wire it as a blocking CI check ("counters match tree"). Today 7 locations disagree, including marketplace.json with itself (says 64, contains 66).
2. **Fix the phantom-path epidemic in root commands (28/39 affected).** Mechanical fix: insert the missing `skills/` segment (`<domain>/skills/<skill>/scripts/…`); add a CI grep that fails on any in-repo path reference in commands/ + agents/ that doesn't exist. Same fix applies to agents' `skills:` shorthand, orchestration/ORCHESTRATION.md examples, and templates.
3. **Reconcile marketplace.json with disk**: register or explicitly de-list the 11 unregistered plugin.json manifests (compliance-os is even advertised in the marketplace description but uninstallable from it); bump the self-described plugin count to the real entry count.
4. **De-duplicate the 6 byte-identical root commands** (karpathy-check + 5 wiki-*) — keep the plugin copies as canonical; cut `prd`, `sprint-plan`, `tdd` or merge them into their skills (bare-prompt-replaceable).
5. **Repair the meta-tooling**: give audit_skills.py and generate-docs.py argparse (`--help` must be side-effect-free — generate-docs.py currently rewrites docs/ on `--help`); add `markdown-html` to sync-hermes/vibe/gemini + convert.sh + generate-docs; extend ci-quality-gate compileall to the 8 uncovered domains.
6. **Fix the template that breeds B1 failures**: add a mandatory trigger-phrase line ("Use when… / Spawn when…") to templates/agent-template.md, then backfill the 17 root agents missing triggers (the 4 v2.8.1 engineering agents are the pattern to copy); update templates/CLAUDE.md's "(when created)" phantom inventory.
### Custom verification criteria (cross-cutting contract)
- `python3 scripts/check_plugin_json.py --all` exits 0 with 77+ manifests OK (keep green).
- `python3 -c "import json; m=json.load(open('.claude-plugin/marketplace.json')); n=len(m['plugins']); assert str(n) in m['metadata']['description'], (n, 'not in metadata')"` exits 0 after counter fix.
- For every `*.py` path matched by `` grep -rhoE '`[a-z-]+(/[a-zA-Z0-9_.-]+)+\.py`' commands/*.md ``: the file exists relative to repo root (0 misses; today: 28 commands fail).
- `python3 scripts/audit_skills.py --help` and `python3 scripts/generate-docs.py --help` exit 0 in <2s with **zero filesystem writes** (`git status --porcelain` empty after).
- `grep -RLE "Use when|Spawn when|Invoke via|Use PROACTIVELY" $(find agents -name 'cs-*.md')` returns empty after trigger backfill.
audit/newgen-2026-06/engineering-team.md
+305
View File
@@ -0,0 +1,305 @@
# Domain audit: engineering-team/ — new-gen model optimization
Audited: 2026-06-10 · Skills: 51 · Agents: 5 · Commands: 0 · Plugins: 6
## Scorecard
| Skill | Verdict | Top issue |
|---|---|---|
| skills/adversarial-reviewer | KEEP | — |
| skills/ai-security | KEEP | — |
| skills/aws-solution-architect | KEEP | cost figures unverified (minor A6) |
| skills/azure-cloud-architect | KEEP | Bicep API versions pinned to 2023 (minor A6) |
| skills/cloud-security | KEEP | — |
| skills/code-reviewer | KEEP | — |
| skills/email-template-builder | OPTIMIZE | 439-line code dump, zero scripts/references (A3, A7) |
| skills/engineering-skills | OPTIMIZE | index skill claims "23 skills" — actual 32; weak as a skill |
| skills/epic-design | KEEP | "You are a world-class expert" filler (minor A2) |
| skills/gcp-cloud-architect | KEEP | — |
| skills/incident-commander | OPTIMIZE | 3 orphan duplicate scripts; SEV taxonomy duplicates incident-response |
| skills/incident-response | KEEP | — |
| skills/ms365-tenant-manager | OPTIMIZE | 3 scripts exist but never referenced in SKILL.md (A3) |
| skills/red-team | KEEP | — |
| skills/security-pen-testing | KEEP | — |
| skills/senior-architect | OPTIMIZE | generic monolith-vs-microservices prose; no verification loop |
| skills/senior-backend | OPTIMIZE | corrupted code snippet (`name: "zstringmin1max100"`) |
| skills/senior-computer-vision | KEEP | — |
| skills/senior-data-engineer | OPTIMIZE | thin body; generic batch-vs-streaming tables (A2) |
| skills/senior-data-scientist | OPTIMIZE | phantom scripts referenced; real 3 scripts orphaned (A3 hard fail) |
| skills/senior-devops | KEEP | — |
| skills/senior-frontend | OPTIMIZE | corrupted snippet (`"cdnexamplecom"`); mid-file generic React dump |
| skills/senior-fullstack | KEEP | — |
| skills/senior-ml-engineer | OPTIMIZE | GPT-4/GPT-3.5/Claude 3 Opus pricing tables (A6 fail) |
| skills/senior-prompt-engineer | REWRITE | entire skill is GPT-4-era prompt engineering; stale models hardcoded in scripts |
| skills/senior-qa | OPTIMIZE | 2 corrupted code snippets; generic RTL cheatsheet content |
| skills/senior-secops | KEEP | trim BAD/GOOD security basics (minor A2) |
| skills/senior-security | CUT-OR-MERGE | duplicates senior-secops/pen-testing/incident-response; no exact CLI for its 2 scripts |
| skills/stripe-integration-expert | OPTIMIZE | pinned `apiVersion: "2024-04-10"`; pure code dump, no tools |
| skills/tdd-guide | KEEP | — |
| skills/tech-stack-evaluator | OPTIMIZE | "ecosystem health from GitHub/npm metrics" is offline static data — staleness unlabeled |
| skills/threat-detection | KEEP | — |
| a11y-audit/skills/a11y-audit | KEEP | — |
| google-workspace-cli/skills/google-workspace-cli | REWRITE | install coordinates almost certainly fabricated (`npm i -g @anthropic/gws`, `github.com/googleworkspace/cli`) |
| snowflake-development/skills/snowflake-development | KEEP | — |
| playwright-pro/skills/pw | KEEP | — |
| playwright-pro/skills/init | KEEP | — |
| playwright-pro/skills/generate | KEEP | — |
| playwright-pro/skills/review | KEEP | — |
| playwright-pro/skills/fix | KEEP | — |
| playwright-pro/skills/migrate | KEEP | — |
| playwright-pro/skills/coverage | KEEP | — |
| playwright-pro/skills/report | KEEP | — |
| playwright-pro/skills/testrail | KEEP | — |
| playwright-pro/skills/browserstack | KEEP | — |
| self-improving-agent/skills/self-improving-agent | KEEP | — |
| self-improving-agent/skills/review | KEEP | — |
| self-improving-agent/skills/promote | KEEP | — |
| self-improving-agent/skills/extract | KEEP | — |
| self-improving-agent/skills/remember | KEEP | — |
| self-improving-agent/skills/status | KEEP | — |
**Totals: KEEP 35 · OPTIMIZE 13 · REWRITE 2 · CUT-OR-MERGE 1**
## Domain-level findings
1. **Bulk-edit code corruption (systemic, 4 confirmed sites).** A past YAML/quoting sweep mangled string literals inside code blocks: `senior-qa/SKILL.md:123` (`name: "click-mei-tobeinthedocument"` — was `getByRole('button', { name: /click me/i })` + `toBeInTheDocument()`), `senior-qa:244` (`"submiti"`), `senior-backend:253` (`name: "zstringmin1max100"` — was `z.string().min(1).max(100)`), `senior-frontend:425` (`hostname: "cdnexamplecom"` — dots stripped). Any model copying these examples emits broken code. Grep pattern to find more: strings that are concatenated identifiers with punctuation stripped.
2. **Stale LLM-era content concentrated in 2 skills + their scripts.** senior-prompt-engineer and senior-ml-engineer present GPT-4/GPT-3.5/Claude 3 Opus model names, 8K context windows, and 2024 pricing as current — in SKILL.md, references (`llm_integration_guide.md`), and hardcoded in scripts (`prompt_optimizer.py` MODEL choices/prices, `agent_orchestrator.py` cost tables). A6 fail across the whole package, not just prose.
3. **Two generations of skills coexist.** The 2026-upgraded trio (senior-fullstack/frontend/backend: decision engines, profiles, forcing questions, composition maps, kill criteria) and the v2.2 security suite (ai-security, threat-detection, incident-response, cloud-security, red-team: exit-code contracts, ATT&CK/ATLAS mapping, anti-patterns) are the new-gen template. The 2025-era role skills (architect, data-scientist, data-engineer, prompt-engineer, qa) still carry "cheatsheet" bodies — generic tables (HTTP status codes, RTL queries, zero-shot vs few-shot) that a frontier model embodies and that cost context for nothing.
4. **Security skill sprawl with duplicated incident-response content.** Four skills carry SEV1-SEV4 frameworks + IR phase checklists (senior-secops, senior-security, incident-response, incident-commander), and OWASP Top 10 appears in 4 places. The v2.2 suite has explicit "this is NOT X" disambiguation tables; the older senior-security does not and is ~80% subsumed.
5. **Orphan/phantom script wiring (A3).** senior-data-scientist references `scripts/train.py`/`evaluate.py`/`health_check.py` (don't exist) while its 3 real scripts go unmentioned; ms365-tenant-manager never references its 3 scripts; incident-commander ships 6 scripts but wires only 3 (`severity_classifier.py`, `incident_timeline_builder.py`, `postmortem_generator.py` are duplicates of the wired ones).
6. **Count drift everywhere.** README says 18 skills, START_HERE says 14, engineering-skills SKILL.md says 23, plugin.json says 32, root CLAUDE.md says 51. engineering-team/CLAUDE.md documents 7 scripts that don't exist under those names (`fullstack_scaffolder.py`, `statistical_analyzer.py`, `etl_generator.py`, `mlops_setup_tool.py`, `llm_integration_builder.py`, `rag_system_builder.py` under prompt-engineer, `video_processor.py`).
7. **18 stale .zip archives at domain root** (senior-*.zip, code-reviewer.zip, etc.) — dead weight shipped to every cloner; almost certainly out of sync with the live folders.
8. **Unverifiable external-tool provenance.** google-workspace-cli teaches a `gws` CLI installed via `npm install -g @anthropic/gws` (not an Anthropic package) with releases at `github.com/googleworkspace/cli` (not a real repo). If the CLI doesn't exist under these coordinates, the entire 373-line skill + 43 recipes is unusable.
9. **Bright spots worth templating:** playwright-pro sub-skills end every workflow with an executable gate ("run `--repeat-each=10`, all 10 must pass"); code-reviewer ships regression fixtures with committed expected `--json` outputs; the security suite's exit-code contracts (0/1/2 with required action) are exactly the A4 pattern the rubric wants.
## Per-skill findings
### engineering-team/skills/senior-prompt-engineer
Verdict: REWRITE
Issues:
- A6 hard fail: GPT-4 cost estimates in sample output (line 57), `--model gpt-4` examples (line 72); `prompt_optimizer.py` restricts `--model` to gpt-4/gpt-3.5-turbo/claude-3-* with 2024 prices; `agent_orchestrator.py` defaults to `model: gpt-4` with stale cost table.
- A2: zero-shot/few-shot/CoT/role-prompting tables and "add format enforcement" guidance are 2023-era basics a frontier model embodies.
- A5 gap: nothing on current practice — structured outputs/tool-use APIs, prompt caching, eval-driven iteration, agent context engineering.
- No verification loop beyond "run both prompts against your eval set" (manual).
Verify (definition of done):
- `grep -rE "gpt-4|gpt-3\.5|claude-3-" skills/senior-prompt-engineer/` returns 0 hits.
- `python3 scripts/prompt_optimizer.py /tmp/p.txt --analyze` exits 0 and model list contains only current-generation model IDs (or is model-agnostic).
- SKILL.md workflow ends with an executable eval gate (script run + exit-code assertion), not "compare outputs".
### engineering-team/google-workspace-cli/skills/google-workspace-cli
Verdict: REWRITE
Issues:
- Install section points to `npm install -g @anthropic/gws` and `github.com/googleworkspace/cli/releases` — neither coordinate is verifiable as real; skill is unusable if the CLI doesn't exist as described.
- All 43 recipes, persona bundles, and command syntax inherit this provenance risk (A5/A6).
- The 5 Python wrappers (`gws_doctor.py` etc.) are fine but only meaningful if `gws` resolves.
Verify (definition of done):
- Documented install command succeeds on a clean machine (`gws --version` exits 0), or the skill is rebuilt around a verifiable tool (GAM7 / Google Workspace Admin SDK + gcloud).
- `python3 scripts/gws_doctor.py` exits non-zero with a clear "gws not installed" message (graceful degradation check).
- 5 randomly sampled recipe commands validated against the CLI's actual `--help` output.
### engineering-team/skills/senior-security
Verdict: CUT-OR-MERGE (fold into senior-secops; keep threat modeling)
Issues:
- ~80% duplicates siblings: incident-response workflow (also in senior-secops + incident-response), secure-code-review checklist (code-reviewer), security headers (senior-secops), tool lists (security-pen-testing).
- A3: scripts section says "see the script source files directly" — no exact CLI invocations for `threat_modeler.py`/`secret_scanner.py`, no output consumption.
- Unique value is only the STRIDE-per-element matrix + DREAD scoring + threat_modeler.py.
Verify (definition of done):
- `python3 scripts/threat_modeler.py --help` exits 0 and the surviving SKILL.md (wherever it lands) shows an exact invocation whose JSON output feeds a named next step.
- After merge, `grep -l "STRIDE" engineering-team/skills/*/SKILL.md` returns exactly one file.
- No SEV/IR phase table remains in the merged body (route to incident-response instead).
### engineering-team/skills/senior-ml-engineer
Verdict: OPTIMIZE
Issues:
- A6: cost table (lines 154-157) lists GPT-4/GPT-3.5/Claude 3 Opus/Haiku at 2024 prices; `references/llm_integration_guide.md` repeats it plus "GPT-4 8,192 context" and `model="gpt-4"` defaults.
- Provider abstraction + tenacity retry code is generic boilerplate (A2).
- Tools shown with one-line CLI but no output-consumption step (`--deploy` flag semantics unstated).
Verify (definition of done):
- `grep -rE "GPT-4|GPT-3\.5|Claude 3 " skills/senior-ml-engineer/` returns 0 hits.
- `python3 scripts/model_deployment_pipeline.py --help` exits 0 and SKILL.md states what artifact each tool emits and which workflow step consumes it.
### engineering-team/skills/senior-data-scientist
Verdict: OPTIMIZE
Issues:
- A3 hard fail: "Common Commands" references `scripts/train.py`, `scripts/evaluate.py`, `scripts/health_check.py` — none exist; the real scripts (`experiment_designer.py`, `feature_engineering_pipeline.py`, `model_evaluation_suite.py`) are never mentioned.
- Body is inline Python a frontier model writes on demand; the embedded checklists (SRM check <0.01, Bonferroni, parallel trends, HC3) are the actual value — keep those, cut the function bodies.
- Generic kubectl/docker/helm command block is irrelevant filler (A7).
Verify (definition of done):
- Every script path in SKILL.md exists: `grep -o "scripts/[a-z_]*\.py" SKILL.md | xargs -I{} test -f skills/senior-data-scientist/{}` all pass.
- `python3 scripts/experiment_designer.py --help` exits 0 and SKILL.md shows an exact invocation per tool.
### engineering-team/skills/senior-qa
Verdict: OPTIMIZE
Issues:
- Corrupted snippets at lines 123 and 244 (mangled `getByRole` calls) — copy-paste hazards.
- RTL query/async/MSW quick-reference is frontier-model-embodied content (A2); MSW example uses deprecated `rest` API (v1) — current msw is `http` (A6).
- `actions/upload-artifact@v3` in CI example is deprecated.
- Coverage workflow is good (threshold + `--strict` exit 1) — keep.
Verify (definition of done):
- All TS/TSX code blocks in SKILL.md parse (extract fenced blocks, run through `tsc --noEmit` or eslint-parse smoke check).
- `python3 scripts/coverage_analyzer.py assets-or-sample --threshold 80` documented and exits per stated contract (0 pass / 1 below threshold).
### engineering-team/skills/senior-frontend
Verdict: OPTIMIZE
Issues:
- Corrupted config at line 425: `remotePatterns: [{ hostname: "cdnexamplecom" }]` (dots stripped).
- Lines 197-465: compound-components/render-props/Image/Suspense dump duplicates what the model knows and what `references/react_patterns.md` already holds — violates progressive disclosure (A2).
- The 2026 wrapper (profiles, decision engine, forcing questions) is excellent; the legacy middle dilutes it.
Verify (definition of done):
- `python3 scripts/frontend_decision_engine.py --primary-device mobile-4g --lcp-target-ms 2000 --seo-dependent true --auth-walled false --team-size 5` exits 0 and emits matched profile + thresholds.
- SKILL.md under 350 lines with pattern code moved to references/; corrupted snippet fixed.
### engineering-team/skills/senior-backend
Verdict: OPTIMIZE
Issues:
- Corrupted Zod snippet at line 253 (`name: "zstringmin1max100"`).
- HTTP status-code table, REST response formats = frontier-embodied filler (A2).
- Load-tester flags shown (`--expect-rate-limit`, `--expect-status`) need verification against actual argparse surface.
Verify (definition of done):
- `python3 scripts/backend_decision_engine.py --team-size 8 --qps-p99 50 --read-write-ratio 20 --tenancy shared-multi-tenant --data-sensitivity pii --pattern modular-monolith --language-preference typescript` exits 0 with profile + SLO floor + approver chain.
- `python3 scripts/api_load_tester.py --help` lists every flag SKILL.md uses.
### engineering-team/skills/senior-architect
Verdict: OPTIMIZE
Issues:
- Monolith-vs-microservices checkboxes and team-size tables are generic (A2) — senior-fullstack's forcing-question + kill-criterion treatment of the same decision is strictly better; cross-link instead of duplicating.
- No verification loop: workflows end at "document decision" (A4).
- Tools are well-wired (exact CLI, sample outputs) — keep.
Verify (definition of done):
- `python3 scripts/dependency_analyzer.py . --output json` exits 0 and emits `circular` + `coupling_score` keys; SKILL.md workflow ends with "re-run analyzer, assert circular = 0".
- ADR step references a concrete template file that exists in the package.
### engineering-team/skills/senior-data-engineer
Verdict: OPTIMIZE
Issues:
- 34-line trigger-phrase section is frontmatter duplication (A2); body's batch-vs-streaming and Lambda-vs-Kappa tables are textbook content.
- "Workflows → See references/workflows.md" and "Troubleshooting → See ..." one-liners make the body a stub while references hold the substance — inverted disclosure (workflows.md is solid at 624 lines).
- Tool subcommand contracts (`generate`/`validate`/`analyze`) shown but outputs not consumed by named steps.
Verify (definition of done):
- `python3 scripts/data_quality_validator.py --help` exits 0 and supports the `validate --checks freshness,completeness,uniqueness` syntax shown.
- SKILL.md inlines a 10-line decision rule per workflow with the deep dive staying in references/.
### engineering-team/skills/incident-commander
Verdict: OPTIMIZE
Issues:
- Orphan scripts: `severity_classifier.py`, `incident_timeline_builder.py`, `postmortem_generator.py` duplicate the 3 wired tools (A3/A7) — delete or wire.
- SEV1-SEV4 definitions overlap incident-response (security flavor) with no disambiguation table; add "this is NOT security incident triage" routing.
- Marketing-prose header block ("battle-tested practices ... at scale") is filler (A2).
Verify (definition of done):
- `ls scripts/ | wc -l` equals the number of scripts referenced in SKILL.md.
- `echo '{"description":"...","affected_users":"80%","business_impact":"high"}' | python3 scripts/incident_classifier.py` exits 0 and output matches `expected_outputs/incident_classification_text_output.txt` semantics.
### engineering-team/skills/ms365-tenant-manager
Verdict: OPTIMIZE
Issues:
- 3 scripts (`powershell_generator.py`, `tenant_setup.py`, `user_management.py`) never referenced in SKILL.md; root-level `sample_input.json`/`expected_output.json` orphaned too (A3).
- PowerShell content is genuinely expert (Graph SDK, CA report-only-first) — keep; just wire or delete the Python layer.
- Verify Graph cmdlet names still current (MSOnline/AzureAD modules retired; this correctly uses Mg* — confirm references do too).
Verify (definition of done):
- Either SKILL.md shows exact CLI for all 3 scripts with consumed output, or `scripts/` is removed.
- `python3 scripts/powershell_generator.py --help` exits 0 (if retained).
### engineering-team/skills/stripe-integration-expert
Verdict: OPTIMIZE
Issues:
- Pinned `apiVersion: "2024-04-10"` presented as current (A6).
- 476 lines of TSX/route-handler code a frontier model writes; the durable value (lifecycle state machine, webhook event ordering, idempotency discipline) should lead, code moved to references/.
- No scripts, no references, no verification loop (A3/A4) — e.g., no webhook-handler checklist gate.
Verify (definition of done):
- `grep -c "apiVersion" SKILL.md` hits use a placeholder + "check current API version" instruction, not a pinned date.
- Workflow ends with executable check: `stripe listen`/`stripe trigger checkout.session.completed` smoke procedure with expected handler behavior stated.
### engineering-team/skills/email-template-builder
Verdict: OPTIMIZE (merge candidate with marketing email skills if it stays code-only)
Issues:
- Entire skill is one 439-line code dump: no scripts/, references/, assets/ (A3/A7).
- React Email component code is frontier-embodied; the value is the pitfalls list (600px, inline styles, dark-mode `!important`, separate sending domains) and provider matrix — invert the ratio.
- No verification loop (no spam-score check step, no preview-render gate) (A4).
Verify (definition of done):
- SKILL.md ≤ 200 lines centered on client-compatibility rules + deliverability checklist; full code in references/.
- Workflow ends with an executable gate (e.g., render template via `npx react-email` preview + checklist assertion of plain-text part present).
### engineering-team/skills/tech-stack-evaluator
Verdict: OPTIMIZE
Issues:
- "Ecosystem health from GitHub, npm metrics" is a stdlib offline tool — data is embedded snapshots with no as-of date; comparisons silently age (A6).
- Quick-start examples are bare prose prompts, not tool invocations — wire them to the scripts.
- 7 scripts but SKILL.md gives exact CLI for only 5 (format_detector, report_generator unmentioned).
Verify (definition of done):
- `python3 scripts/tco_calculator.py --input assets/sample_input_tco.json` exits 0 and matches `assets/expected_output_comparison.json` schema.
- Every embedded-data script prints a `data_as_of` field in JSON output and SKILL.md tells the model to label conclusions with it.
### engineering-team/skills/engineering-skills
Verdict: OPTIMIZE
Issues:
- Claims "23 production-ready engineering skills"; `skills/` holds 32; plugin.json says 32; README says 18; START_HERE says 14 — pick one truth (A6/E2).
- As a skill it's a static catalog; its only durable instruction ("load one SKILL.md, don't bulk-load") could live in the plugin description.
- `npx agent-skills-cli add ...` install path needs verification.
Verify (definition of done):
- `ls -d engineering-team/skills/*/ | wc -l` equals the count stated in SKILL.md, plugin.json, README.md, and START_HERE.md.
- Skill table lists every actual folder (no missing security-suite rows).
## KEEP-verdict verification criteria
- **adversarial-reviewer** — review output contains all 3 persona sections each with ≥1 finding and ends with verdict ∈ {BLOCK, CONCERNS, CLEAN}; promotion rule applied when 2+ personas overlap.
- **ai-security** — `python3 scripts/ai_threat_scanner.py --target-type llm --access-level black-box --json` exits 0/1/2 per contract and JSON has `overall_risk` + `findings[].finding_type`; `--access-level gray-box` without `--authorized` exits 2.
- **aws-solution-architect** — `python3 scripts/architecture_designer.py --input <sample>` emits `recommended_pattern` + `estimated_monthly_cost_usd`; CloudFormation output passes `aws cloudformation validate-template` (or cfn-lint) when run.
- **azure-cloud-architect** — `python3 scripts/bicep_generator.py --arch-type web-app --output /tmp/main.bicep` exits 0 and output parses with `az bicep build` where available.
- **cloud-security** — `python3 scripts/cloud_posture_check.py <iam-sample>.json --check iam --json` exits 2 on a PassRole+CreateFunction policy and 0 on least-privilege sample.
- **code-reviewer** — `python3 scripts/code_quality_checker.py assets/sample_java_smells.java --json | diff - expected_outputs/sample_java_smells_quality.json` is empty (regression fixture green); dispatch table covers every `languages/*.md` file.
- **epic-design** — `python3 scripts/inspect-assets.py --help` exits 0 without Pillow installed; output format matches references/asset-pipeline.md Step 4 contract.
- **gcp-cloud-architect** — `python3 scripts/cost_optimizer.py --resources <sample>.json --monthly-spend 2000` exits 0 with itemized savings.
- **incident-response** — `echo '{"event_type":"ransomware","host":"x","raw_payload":{}}' | python3 scripts/incident_triage.py --classify --json` exits 2 (SEV1) and maps T1486.
- **red-team** — `python3 scripts/engagement_planner.py --techniques T1059 --access-level external --json` (no `--authorized`) exits 1; with `--authorized` exits 0 and orders phases by kill chain.
- **security-pen-testing** — `python3 scripts/vulnerability_scanner.py --target web --scope quick --json` exits 0 and emits OWASP A01-A10 checklist items; `dependency_auditor.py --file package.json --json` parses.
- **senior-computer-vision** — `python3 scripts/dataset_pipeline_builder.py --help` lists `--analyze/--clean/--split/--generate-config` flags used in SKILL.md; `inference_optimizer.py --help` lists `--benchmark/--export`.
- **senior-devops** — `python3 scripts/terraform_scaffolder.py /tmp/infra --provider=aws --module=ecs-service` exits 0; generated HCL passes `terraform validate` where available; rollback procedure's `curl -sf .../healthz` gate retained.
- **senior-fullstack** — `python3 scripts/fullstack_decision_engine.py --sample --output json` exits 0 with `ranked_matches[0].profile_name` and empty `kill_criteria_tripped` (verified this audit); engine refuses (non-zero) when any of the 4 required inputs missing.
- **senior-secops** — `python3 scripts/security_scanner.py <dir>` exit codes follow 0/1/2 contract; `compliance_checker.py --framework soc2 --json` emits per-control results; CVE-triage SLA table retained (9.0+ internet-facing = 24h).
- **tdd-guide** — `python3 scripts/coverage_analyzer.py --report assets/sample_coverage_report.lcov --threshold 80` exits per contract and P0/P1/P2 buckets present; `test_generator.py --input <py> --framework pytest` output compiles.
- **threat-detection** — `python3 scripts/threat_signal_analyzer.py --mode anomaly --events-file <sample> --baseline-mean 100 --baseline-std 25 --json` exits 2 only when z ≥ 3.0; IOC mode flags >30-day-old IPs as `stale`.
- **a11y-audit** — `python3 scripts/contrast_checker.py --fg "#777777" --bg "#ffffff"` reports fail at 4.5:1; `a11y_scanner.py <dir> --ci` exits non-zero only on critical findings; `--baseline` comparison runs.
- **snowflake-development** — `python3 scripts/snowflake_query_helper.py merge --target t --source s --key id --columns a,b` exits 0 and emitted SQL contains colon-prefix rule compliance in proc templates.
- **playwright-pro/pw** — all 9 sub-skill routes listed exist as skill folders; Quick Start sequence references only shipped commands.
- **playwright-pro/init** — generated `playwright.config.ts` sets `retries: 2` in CI / `0` local and `trace: 'on-first-retry'`; first smoke test runs.
- **playwright-pro/generate** — workflow step 7 retained: `npx playwright test <file> --reporter=list` must run before reporting done; no `waitForTimeout` in output.
- **playwright-pro/review** — review loads `anti-patterns.md` (file exists) and flags a seeded `page.waitForTimeout()` fixture.
- **playwright-pro/fix** — fix loads `flaky-taxonomy.md` (file exists); completion requires `--repeat-each=10` 10/10 green.
- **playwright-pro/migrate** — post-migration step routes to `/pw:coverage` parity check before decommissioning old suite.
- **playwright-pro/coverage** — output lists tested vs untested routes with priority ranking.
- **playwright-pro/report** — report generation consumes `playwright-report/` or JSON reporter output, errors clearly when absent.
- **playwright-pro/testrail** — refuses gracefully with setup instructions when `TESTRAIL_URL/USER/API_KEY` unset.
- **playwright-pro/browserstack** — refuses gracefully when `BROWSERSTACK_USERNAME/ACCESS_KEY` unset.
- **self-improving-agent (root)** — memory-paths table matches current Claude Code memory layout (`~/.claude/projects/<path>/memory/MEMORY.md`, 200-line load) — recheck against docs each release.
- **si/review** — spawns memory-analyst; output buckets = promotion candidates / stale / consolidation / conflicts / health.
- **si/promote** — promotion writes to CLAUDE.md or `.claude/rules/` AND removes the MEMORY.md source entry (both halves verified).
- **si/extract** — extracted skill passes `scripts/audit_skills.py` (repo validator) with no FAIL.
- **si/remember** — entry written with timestamp + category into the active memory dir.
- **si/status** — dashboard reports line counts vs 200-line budget and flags overflow topic files.
## Agents
| Agent | B1 frontmatter | B2 differentiation | B3 body | Verdict |
|---|---|---|---|---|
| playwright-pro/agents/test-architect | PASS (read-only tools) | PASS — plans, explicitly does not write tests | PASS | KEEP |
| playwright-pro/agents/test-debugger | PASS — exemplary scoped `Bash(npx playwright test *)` allowlist + disallowedTools | PASS — taxonomy-driven diagnosis | PASS | KEEP |
| playwright-pro/agents/migration-planner | PASS (read-only) | PASS — detection protocol per framework | PASS | KEEP |
| self-improving-agent/agents/memory-analyst | PASS (Read/Glob/Grep, maxTurns 30) | PASS — read-only analyst, distinct outputs | PASS | KEEP |
| self-improving-agent/agents/skill-extractor | PASS (Write/Edit + disallowedTools) | PASS — portability rules (no hardcoded paths) | PASS | KEEP |
Note: agent descriptions use "Invoked by /pw:..." rather than "Use when..." trigger phrasing — acceptable since they are command-spawned, not auto-triggered.
## Commands
None in scope. engineering-team/ ships no `commands/` directory; the `/pw:*` and `/si:*` surfaces are skills-as-commands (audited above). The grep hits under `commands*` are reference docs, not slash commands.
## Plugin manifests
| Plugin | Issue |
|---|---|
| `.claude-plugin/plugin.json` (engineering-skills) | Says "32 skills" — matches `skills/` dir count, but conflicts with SKILL.md (23), README (18), START_HERE (14). Description is a 1,100-char wall; trim. |
| `a11y-audit` | Clean; description matches contents. |
| `google-workspace-cli` | Description repeats the unverifiable `gws` CLI claims; fix alongside the skill REWRITE. |
| `playwright-pro` (name: `pw`) | Clean; "55+ templates, 3 agents" — template count not verified file-by-file but folder structure exists. |
| `self-improving-agent` (name: `si`) | Clean; commands listed all exist as sub-skills. |
| `snowflake-development` | Description's "query helper script, 3 reference guides" verified accurate (1 script, 3 refs). Clean. |
Additional manifest-adjacent debt: 18 stale `.zip` archives at engineering-team root; engineering-team/CLAUDE.md documents 7 nonexistent script filenames and a "32 skills / 39+ tools" inventory that predates the security suite and sub-plugins.
audit/newgen-2026-06/engineering.md
+333
View File
@@ -0,0 +1,333 @@
# Domain audit: engineering/ — new-gen model optimization
Audited: 2026-06-10 · Skills: 80 SKILL.md (63 distinct skills; 12 sub-command skills under agenthub/autoresearch; 4 dual-published duplicates; 1 sample asset excluded) · Agents: 14 · Commands: 14 · Plugins: 28 manifests
## Scorecard
Bundle = `engineering/skills/<name>`; standalone = `engineering/<name>/`.
| Skill | Verdict | Top issue |
|---|---|---|
| skills/agent-designer | REWRITE | 279 lines of taxonomy prose a frontier model already knows; 3 root-level scripts never wired |
| skills/agent-workflow-designer | OPTIMIZE | Thin body; overlaps agent-designer + workflow-builder |
| skills/api-design-reviewer | OPTIMIZE | ~300 lines of textbook REST; scripts named but no exact CLI in workflow |
| skills/api-test-suite-builder | KEEP | — |
| skills/browser-automation | KEEP | — |
| skills/changelog-generator | KEEP | — (absorb release-manager into it) |
| skills/chaos-engineering (+ standalone dup) | KEEP | — (deduplicate copies) |
| skills/ci-cd-pipeline-builder | KEEP | — |
| skills/codebase-onboarding | OPTIMIZE | Thin; 1 script, low expertise density |
| skills/command-guide | CUT-OR-MERGE | Documents another repo's (ECC) commands/agents that don't exist here |
| skills/database-designer | CUT-OR-MERGE | Overlaps 2 siblings; claims "included tools" with zero CLI wiring |
| skills/database-schema-designer | CUT-OR-MERGE | No scripts; broken seed-code example; overlaps database family |
| skills/dependency-auditor | REWRITE | Marketing brochure ("Future Enhancements", "Planned Features"); unverifiable claims |
| skills/engineering-advanced-skills | OPTIMIZE | Index says "25 skills", plugin says 40; wrong load paths |
| skills/env-secrets-manager | KEEP | — (fix dead cross-refs) |
| skills/feature-flags-architect (+ dup) | KEEP | — (deduplicate copies) |
| skills/focused-fix | KEEP | — (references external `superpowers:*` skills) |
| skills/full-page-screenshot | KEEP | — |
| skills/git-worktree-manager | KEEP | — |
| skills/interview-system-designer | OPTIMIZE | HR skill in engineering domain; 4 scripts, 1 wired |
| skills/kubernetes-operator (+ dup) | KEEP | — (deduplicate copies) |
| skills/mcp-server-builder | KEEP | — |
| skills/migration-architect | REWRITE | 477 lines textbook; scripts named only in "Tools" section, no CLI |
| skills/monorepo-navigator | KEEP | — |
| skills/observability-designer | REWRITE | Brochure prose; no exact CLI; overlaps slo-architect |
| skills/performance-profiler | KEEP | — |
| skills/pr-review-expert | KEEP | — |
| skills/rag-architect | REWRITE | Stale (ada-002, 2024 Pinecone pricing); 3 scripts never wired; textbook |
| skills/release-manager | CUT-OR-MERGE | 489-line textbook; duplicates changelog-generator; scripts unwired |
| skills/runbook-generator | OPTIMIZE | Thin skeleton generator, low expertise density |
| skills/secrets-vault-manager | KEEP | — |
| skills/self-eval | KEEP | — |
| skills/ship-gate | KEEP | — |
| skills/skill-security-auditor | KEEP | — |
| skills/skill-tester | REWRITE | 390-line brochure incl. "Future Enhancements"; CLI shown without paths |
| skills/slo-architect (+ dup) | KEEP | — (deduplicate copies) |
| skills/spec-driven-workflow | KEEP | — |
| skills/sql-database-assistant | KEEP | — (merge target for the database trio) |
| skills/tc-tracker | KEEP | — |
| skills/tech-debt-tracker | REWRITE | Roadmap/KPI filler; 5 passing scripts, zero CLI wiring |
| agenthub (8 SKILL.md) | KEEP | — |
| autoresearch-agent (6 SKILL.md) | KEEP | — (document evaluator --help exception in SKILL.md) |
| behuman | KEEP | — (not registered in marketplace) |
| caveman | KEEP | — |
| claude-coach | OPTIMIZE | Duplicate frontmatter keys; README content pasted into SKILL.md tail |
| code-tour | KEEP | — |
| data-quality-auditor | KEEP | — |
| demo-video | KEEP | — |
| docker-development | KEEP | — |
| grill-me | KEEP | — |
| grill-with-docs | KEEP | — (not registered in marketplace) |
| handoff (engineering) | KEEP | — |
| helm-chart-builder | KEEP | — |
| karpathy-coder | KEEP | — |
| llm-cost-optimizer | KEEP | — (not registered in marketplace) |
| llm-wiki | KEEP | — |
| prompt-governance | KEEP | — (not registered in marketplace) |
| security-guidance | KEEP | — |
| statistical-analyst | KEEP | — |
| terraform-patterns | KEEP | — |
| universal-scraping-architect | OPTIMIZE | 3 orphan scripts; placeholder agent + command; non-stdlib deps |
| workflow-builder | KEEP | — |
| write-a-skill | KEEP | — |
**Totals: KEEP 44 · OPTIMIZE 8 · REWRITE 7 · CUT-OR-MERGE 4** (63 distinct skills)
## Domain-level findings
1. **Two clear generations.** v2.4+ skills (slo-architect, chaos-engineering, kubernetes-operator, feature-flags-architect, karpathy-coder, workflow-builder, the Pocock ports, agenthub, autoresearch) are exemplary: trigger-rich descriptions, exact CLI per tool, refusal gates, "Verifiable success" sections. ~10 v2.0-era skills in `engineering/skills/` are capability brochures ("Future Enhancements", "Conclusion", "Planned Features" sections) whose body is knowledge a frontier model already has — pure context dead weight.
2. **Orphan-script epidemic in v2.0-era skills.** agent-designer, rag-architect, release-manager, database-designer (root-level `.py`), tech-debt-tracker and skill-tester (in `scripts/`) all ship working scripts (`--help` passes) that SKILL.md never invokes with a runnable command. A model following the SKILL.md will never run them — A3 failure across the board. universal-scraping-architect same pattern.
3. **Overlap clusters burning context.** (a) Database trio: database-designer / database-schema-designer / sql-database-assistant — sql-database-assistant alone covers ~90%; (b) release pair: release-manager vs changelog-generator (changelog-generator is the wired, lean one); (c) observability-designer vs slo-architect (slo-architect is strictly better on the SLO half).
4. **Byte-identical dual-published copies.** slo-architect, chaos-engineering, kubernetes-operator, feature-flags-architect each exist in both `engineering/skills/` and `engineering/<name>/skills/<name>/` (verified `diff` identical). No single source of truth; edits will diverge.
5. **Counter and registry drift.** Bundle index SKILL.md says "25 advanced engineering skills"; its plugin.json says 40; the marketplace description for `engineering-advanced-skills` lists skills (llm-cost-optimizer, prompt-governance, behuman, code-tour, demo-video, data-quality-auditor, statistical-analyst, llm-wiki…) that live in standalone plugin folders OUTSIDE the manifest's `"skills": ["./skills"]` path. Separately, 5 plugins with valid plugin.json (behuman, claude-coach, grill-with-docs, llm-cost-optimizer, prompt-governance) are not registered in marketplace.json at all.
6. **Dead cross-references.** env-secrets-manager points to `engineering/senior-secops`, `engineering/infrastructure-as-code`, `engineering/container-orchestration` (none exist; senior-secops lives in engineering-team/); sql-database-assistant points to nonexistent `observability-platform`; focused-fix references external `superpowers:*` skills; command-guide is entirely about a foreign ecosystem.
7. **Freshness spots.** rag-architect: `text-embedding-ada-002` as "quality model", "$70/month Pinecone 1M vectors" — 2023/24 facts presented as current. command-guide: `/fast` "(Opus 4.6 only)". api-design-reviewer example timestamps "2024-…".
8. **Reference quality is bimodal.** v2.0-era references (rag-architect, dependency-auditor, agent-designer) are uncited encyclopedic prose a frontier model regenerates on demand; v2.6+ references cite canon by name (Evans/Nygard/Google SRE Workbook/Pocock).
9. **By-design script exceptions partly documented.** autoresearch evaluators carry "DO NOT MODIFY — fixed evaluator" headers, but the SKILL.md never states they intentionally fail `--help`; security-guidance hook's stdin contract IS documented in its SKILL.md (good).
## Per-skill findings
### engineering/skills/agent-designer
Verdict: REWRITE
Issues:
- Entire 279-line body is generic multi-agent taxonomy (Supervisor/Swarm/Pipeline pros-cons) — A2/A5 fail; a frontier model knows all of it.
- 3 working scripts (`agent_planner.py`, `tool_schema_generator.py`, `agent_evaluator.py`, all pass `--help`) are never mentioned in SKILL.md — A3 fail; assets/expected_outputs unused.
- Description is bare trigger sentence with no mention of tools.
- Overlaps agent-workflow-designer and workflow-builder.
Verify: `python3 engineering/skills/agent-designer/agent_planner.py --help` exits 0 AND SKILL.md contains the literal string `agent_planner.py` with a runnable invocation; SKILL.md < 150 lines; `grep -c "Pros:" SKILL.md` returns 0.
### engineering/skills/agent-workflow-designer
Verdict: OPTIMIZE
Issues:
- 83-line body is mostly headers; pattern map duplicates `references/workflow-patterns.md` one-liners.
- No verification loop — scaffolder output is never validated by a named next step.
- Scope collision with workflow-builder (Claude Code Workflow tool) and agent-designer; needs explicit "NOT for" routing.
Verify: `python3 scripts/workflow_scaffolder.py sequential --name t` exits 0 and emits JSON; SKILL.md gains a "When NOT to use" block naming workflow-builder.
### engineering/skills/api-design-reviewer
Verdict: OPTIMIZE
Issues:
- Lines 41–333 restate REST conventions/pagination/status codes any frontier model knows — cut to references or delete.
- Tools section describes features but the only invocations are inside CI YAML examples; no first-class Quick Start CLI.
- Example timestamp `2024-02-16` (A6 nit).
Verify: `python3 scripts/api_linter.py --help` exits 0; SKILL.md has a Quick Start with all 3 script invocations; body ≤ 200 lines.
### engineering/skills/codebase-onboarding
Verdict: OPTIMIZE
Issues:
- 84 lines, single analyzer script; "Tailor output depth by audience" is the only non-obvious content.
- No verification loop (generated doc never validated against repo facts).
Verify: `python3 scripts/codebase_analyzer.py . --json` exits 0 and emits JSON with language/file-count keys; SKILL.md adds a post-generation check (e.g. "every setup command in the doc was executed once").
### engineering/skills/command-guide
Verdict: CUT-OR-MERGE
Issues:
- Documents commands/agents from the ECC ecosystem (`planner`, `build-error-resolver`, `tdd-guide`, `/build-fix`, `/learn`, `/remember`) — none ship in this repo; actively misleads the model into invoking nonexistent tools.
- `/fast` "(Opus 4.6 only)" — stale model gating (A6).
- Zero scripts, zero references; auto-trigger table tells the model to "immediately invoke" agents that don't exist here.
Verify (if kept at all): every command/agent named in the file resolves to a file in this repo (`grep -o '/[a-z-]*' SKILL.md` cross-checked against commands/); otherwise delete from plugin.
### engineering/skills/database-designer
Verdict: CUT-OR-MERGE (fold unique tables into sql-database-assistant)
Issues:
- "The included tools automate common analysis" but no script invocation anywhere; 3 root scripts orphaned (A3).
- JOIN/CTE/window-function content is textbook (A2); decision matrices duplicate sql-database-assistant's.
- Three-way overlap with database-schema-designer and sql-database-assistant; cross-refs to both admit it.
Verify: after merge, `engineering/skills/sql-database-assistant/SKILL.md` contains the sharding/replication tables; `schema_analyzer.py`/`index_optimizer.py`/`migration_generator.py` either wired into sql-database-assistant or deleted.
### engineering/skills/database-schema-designer
Verdict: CUT-OR-MERGE
Issues:
- No scripts at all; SKILL.md is one worked example + RLS snippets.
- Seed-data example is syntactically broken (`name: "fakercompanycatchphrase"` — missing comma, dead faker call, line ~155).
- ERD/normalization mandate duplicates database-designer's claims.
Verify: RLS policy block and pitfalls table migrated into the surviving database skill; broken seed example deleted or fixed to parse with `npx tsc --noEmit`.
### engineering/skills/dependency-auditor
Verdict: REWRITE
Issues:
- ~250 of 337 lines are brochure ("Use Cases & Applications", "Future Enhancements", "Metrics & KPIs") — A2/A7 fail.
- Claims "built-in vulnerability database with 500+ CVE patterns", live "PyPI/npm advisory" cross-referencing — SKILL.md's own scripts are offline pattern matchers; over-claims capability.
- Quick Start has 3 CLI lines buried at the bottom; no output-consumption step, no verification loop.
Verify: `python3 scripts/dep_scanner.py --help` exits 0; rewritten SKILL.md ≤ 150 lines, leads with the 3 CLIs + JSON keys consumed; no "Future Enhancements"/"Planned Features" headings remain.
### engineering/skills/engineering-advanced-skills
Verdict: OPTIMIZE
Issues:
- H1/body says "25 advanced engineering skills"; plugin.json says 40; folder has 39 + index — three different counts.
- Quick Start path `/read engineering/agent-designer/SKILL.md` is wrong (real path `engineering/skills/agent-designer/SKILL.md`).
- Table lists 25 of 39 skills; missing the reliability quartet, ship-gate, self-eval, tc-tracker, etc.
Verify: `ls engineering/skills | wc -l` matches the count stated in SKILL.md and plugin.json description; every path in the table resolves.
### engineering/skills/interview-system-designer
Verdict: OPTIMIZE
Issues:
- Hiring-process skill living in the engineering plugin — domain misfit (product-team/c-level fit better).
- 4 scripts present, only `interview_planner.py` wired; 59-line body with generic best practices.
Verify: all shipped scripts referenced with exact CLI in SKILL.md or removed; `python3 scripts/interview_planner.py --role "SWE" --level senior --json` exits 0 with JSON.
### engineering/skills/migration-architect
Verdict: REWRITE
Issues:
- 477 lines; Strangler Fig/CDC/blue-green content is textbook (A2); "Communication Templates" and "Success Metrics" sections are filler (A7).
- Scripts (`migration_planner.py`, `compatibility_checker.py`, `rollback_generator.py`) appear only as bullet names + one CI YAML snippet — no Quick Start CLI (A3).
- No verification loop; checklists are prose, not machine-checkable.
Verify: `python3 engineering/skills/migration-architect/migration_planner.py --help` exits 0; rewritten SKILL.md ≤ 200 lines with all 3 CLIs and a "plan must pass compatibility_checker with 0 CRITICAL" gate.
### engineering/skills/observability-designer
Verdict: REWRITE
Issues:
- 268 lines of golden-signals/RED/USE/three-pillars prose — pure frontier-model knowledge (A2/A5).
- "Scripts Overview" describes I/O shapes but gives zero runnable commands (A3); scripts pass `--help`.
- Overlaps slo-architect (which does the SLO half with thresholds + math + refusal gates); this skill should shrink to dashboards + alert-noise tooling and route SLO work to slo-architect.
Verify: `python3 scripts/slo_designer.py --help`, `alert_optimizer.py --help`, `dashboard_generator.py --help` all exit 0 AND appear as exact CLIs in SKILL.md; "When NOT to use → slo-architect" block present.
### engineering/skills/rag-architect
Verdict: REWRITE
Issues:
- Stale facts as current: `text-embedding-ada-002` as the quality tier, "Pinecone $70/month for 1M vectors", model lists from 2023/24 (A6).
- 3 root scripts (`chunking_optimizer.py`, `rag_pipeline_designer.py`, `retrieval_evaluator.py`, all pass `--help`) never referenced in SKILL.md (A3).
- 318 lines of chunking/retrieval taxonomy a frontier model knows; only 1 reference doc, uncited (A7).
Verify: `python3 engineering/skills/rag-architect/chunking_optimizer.py --help` exits 0 AND is invoked in SKILL.md; zero occurrences of `ada-002` / hardcoded vendor prices; body ≤ 200 lines.
### engineering/skills/release-manager
Verdict: CUT-OR-MERGE (into changelog-generator)
Issues:
- 489-line SemVer/Git-Flow/conventional-commits textbook (A2); changelog-generator already ships the wired, lean version of the changelog/bump core.
- 3 root scripts named in "Key Components" but never invoked (A3).
- Hotfix SLAs and rollback triggers are the only practitioner content — migrate those tables.
Verify: hotfix-severity and rollback-trigger tables present in the surviving skill; `version_bumper.py`/`release_planner.py` wired with exact CLI or deleted; no duplicate conventional-commit spec across the two skills.
### engineering/skills/runbook-generator
Verdict: OPTIMIZE
Issues:
- 76 lines, one template-skeleton script, generic best practices; weakest of the wired DevOps set.
- No verification loop (runbook never validated — e.g., "every command block has an expected-output check").
Verify: `python3 scripts/runbook_generator.py payments-api --owner x` exits 0 and emits the standard sections; SKILL.md adds a post-generation checklist the model executes (rollback section non-empty, every step has a verify line).
### engineering/skills/skill-tester
Verdict: REWRITE
Issues:
- 390 lines, heavy brochure: "Performance & Scalability", "Security & Safety", "Future Enhancements", "Conclusion" (A7).
- CLI examples lack paths (`skill_validator.py path/to/skill` won't run from repo root); one example references nonexistent `trend_analyzer.py` (phantom script, A3).
- Tier line-count requirements conflict with write-a-skill's "SKILL.md under 100 lines" doctrine — repo-internal contradiction.
Verify: `python3 engineering/skills/skill-tester/scripts/skill_validator.py engineering/skills/self-eval --json` exits 0 with JSON; no reference to `trend_analyzer.py`; body ≤ 200 lines.
### engineering/skills/tech-debt-tracker
Verdict: REWRITE
Issues:
- Body is a 6-week "Implementation Roadmap" + aspirational KPIs ("25% reduction in debt interest rate") — filler, zero operational instructions (A2/A5).
- 5 scripts incl. `debt_scanner.py`/`debt_prioritizer.py`/`debt_dashboard.py` all pass `--help` but SKILL.md contains not one CLI invocation (A3) — worst wiring gap in the domain relative to tooling quality.
- 4 references + 4 assets unreferenced from the body (A7).
Verify: SKILL.md Quick Start runs all 3 core scripts with exact flags; `python3 scripts/debt_scanner.py --help` exits 0; scan→prioritize→dashboard pipeline shows which JSON keys flow between steps.
### engineering/claude-coach
Verdict: OPTIMIZE
Issues:
- Frontmatter has duplicate/case-variant keys (`Name:` + `name:`, `Version: 1.0.0` + `version: 2.9.0`, stray `Tier/Category/Dependencies`) — undefined parse behavior.
- Lines 145–205 re-paste Name/Description/Features/Usage (README content) after the body ends — duplication (A7).
- Scripts listed at the very bottom with no CLI; `coach_tip_classifier.py` is core to Rule 5 but never invoked.
Verify: `python3 -c "import yaml,io; yaml.safe_load(open('engineering/claude-coach/skills/claude-coach/SKILL.md').read().split('---')[1])"` yields exactly one `name`/`version`; `python3 scripts/coach_tip_classifier.py --help` exits 0 and appears as a CLI in the body.
### engineering/universal-scraping-architect
Verdict: OPTIMIZE
Issues:
- 3 scripts (`validate_extraction.py`, `firecrawl_example.py`, `local_bs4_example.py`) never referenced in SKILL.md (A3).
- Agent (`cs-scraping-architect.md`, 6 lines) and command (`cs-scrape.md`, 6 lines) are placeholders — B3/C3 fail.
- "You are an expert…" opener (A2 filler); non-stdlib deps (firecrawl/pandas/bs4) acceptable (BYOK documented) but should be listed per-script.
- Layout anomaly: only engineering plugin with SKILL.md at plugin root (no `skills/` dir).
Verify: `python3 engineering/universal-scraping-architect/scripts/validate_extraction.py --help` exits 0 and is invoked in SKILL.md step 4 ("Validate & Clean"); agent file ≥ 40 lines with tools + triggers or deleted.
## KEEP-verdict verification criteria
- **api-test-suite-builder** — Next.js route-scan command from SKILL.md runs against a sample app dir without error; auth matrix table retains all 6 rows.
- **browser-automation** — `python3 scripts/anti_detection_checker.py --help` exits 0; all 3 referenced reference files exist.
- **changelog-generator** — `printf 'feat: x\nfix: y\n' | python3 scripts/generate_changelog.py --next-version v1.0.0 --format json` exits 0 with `Added`/`Fixed` sections; `commit_linter.py --strict` exits non-zero on `bad message`.
- **chaos-engineering** — `python3 scripts/blast_radius_calculator.py --traffic-share 0.05 --user-pop 1000000 --duration-min 15` exits 0, output contains GREEN/YELLOW/RED; bundle and standalone copies stay byte-identical (`diff -r`) until deduped.
- **ci-cd-pipeline-builder** — `python3 scripts/stack_detector.py --repo . --format json` exits 0 with detected-language keys; generated YAML parses (`python3 -c "import yaml,sys; yaml.safe_load(open('out.yml'))"`).
- **env-secrets-manager** — `python3 scripts/env_auditor.py . --json` exits 0 with severity-tagged findings; cross-reference table contains no path that fails `ls`.
- **feature-flags-architect** — `python3 scripts/rollout_planner.py --population 100000 --target-percent 100 --duration-days 14 --strategy ring` exits 0 with a phased table; `kill_switch_audit.py --help` exits 0.
- **focused-fix** — 5-phase headings (SCOPE/TRACE/DIAGNOSE/FIX/VERIFY) and the 3-strike escalation rule remain; `superpowers:` references either resolve or are reworded as optional externals.
- **full-page-screenshot** — `node scripts/full-page-screenshot.mjs --check` exits with documented status; anti-pattern table intact.
- **git-worktree-manager** — `python3 scripts/worktree_manager.py --help` and `worktree_cleanup.py --help` exit 0; validation checklist (ports file, env copy) retained.
- **kubernetes-operator** — `python3 scripts/crd_validator.py --help` exits 0; capability levels L1–L5 retained; dedupe with bundle copy.
- **mcp-server-builder** — `python3 scripts/openapi_to_mcp.py --help` and `mcp_validator.py --help` exit 0; strict mode returns non-zero on a manifest with a duplicate tool name.
- **monorepo-navigator** — `python3 scripts/monorepo_analyzer.py . --json` exits 0; pitfalls table keeps the `--filter` and `git filter-repo` rows.
- **performance-profiler** — `python3 scripts/performance_profiler.py . --json` exits 0; before/after template and "Measure First" rule retained.
- **pr-review-expert** — security-scan grep block runs against a sample diff without syntax errors; 30+ item checklist count ≥ 30.
- **secrets-vault-manager** — 3 tool names in the Tools table map to existing files in scripts/ (add exact CLI when touched); Vault HCL snippets parse visually.
- **self-eval** — composite matrix unchanged (Low ambition caps at 2; 5 requires High+Strong); scores append to `.self-eval-scores.jsonl`.
- **ship-gate** — `references/checks.md` and `references/patterns.md` exist; category table sums (55 auto + 27 manual) match checks.md entries.
- **skill-security-auditor** — `python3 scripts/skill_security_auditor.py engineering/skills/self-eval --json` exits 0 with verdict ∈ {PASS, WARN, FAIL}.
- **slo-architect** — `python3 scripts/error_budget_calculator.py --target 99.9 --window-days 30` exits 0 and prints 43.20 min allowed downtime (verified this audit); `slo_review.py` flags `target ≥ 99.99` docs.
- **spec-driven-workflow** — `python3 scripts/spec_validator.py --help` and `test_extractor.py --help` exit 0; Iron Law + bounded-autonomy STOP list retained.
- **sql-database-assistant** — `python3 scripts/query_optimizer.py --query "SELECT * FROM t" --dialect postgres` exits 0 with findings; dialect table retains all 4 engines.
- **tc-tracker** — `python3 scripts/tc_init.py --project T --root /tmp/tc-test && python3 scripts/tc_validator.py --registry /tmp/tc-test/docs/TC/tc_registry.json` both exit 0; state machine rejects `planned → deployed`.
- **agenthub** — `python3 scripts/hub_init.py --help`, `dag_analyzer.py --help`, `result_ranker.py --help` exit 0; all 7 `/hub:*` sub-skills name a script or Agent-tool call.
- **autoresearch-agent** — `python3 scripts/run_experiment.py --help` exits 0; evaluators intentionally fail `--help` (fixed contract) — add one sentence to SKILL.md documenting this; `setup_experiment.py --help` exits 0.
- **behuman** — Show/Quiet mode contract + 3 worked examples retained; token-cost table present. Register in marketplace or document why not.
- **caveman** — Matt's persistence + auto-clarity rules verbatim; `python3 scripts/caveman_lint.py "Sure! I'd be happy to help" ` flags filler.
- **code-tour** — schema block contains `$schema: https://aka.ms/codetour-schema`; validation checklist (verified line numbers, ≤2 content steps) intact.
- **data-quality-auditor** — `python3 scripts/data_profiler.py --help`, `missing_value_analyzer.py --help`, `outlier_detector.py --help` all exit 0; DQS weights sum to 100%.
- **demo-video** — fallback ladder (MCPs → manual build.sh) retained; output artifact list (scenes/, narration/, scenes.json, build.sh) unchanged.
- **docker-development** — `python3 scripts/dockerfile_analyzer.py --help` and `compose_validator.py --help` exit 0; 3 multi-stage patterns + base-image decision tree retained.
- **grill-me / grill-with-docs** — one-question-per-turn + recommended-answer rules verbatim; `python3 scripts/context_md_linter.py --help` (grill-with-docs) exits 0. Register grill-with-docs in marketplace.
- **handoff** — `mktemp` convention + no-duplication rule verbatim; 5 sections list unchanged.
- **helm-chart-builder** — `python3 scripts/chart_analyzer.py --help` exits 0; scaffold tree includes pdb.yaml + networkpolicy.yaml.
- **karpathy-coder** — `python3 scripts/complexity_checker.py --help` and `diff_surgeon.py --help` exit 0; 4 principles + relax conditions retained.
- **llm-cost-optimizer** — ROI-ordered 6 techniques with % ranges retained; proactive-flag table (max_tokens unset, >2k-token system prompt) intact; model-tier examples refreshed when model names rotate.
- **llm-wiki** — `python3 scripts/init_vault.py --help` and `lint_wiki.py --help` exit 0; Iron rule (never write raw/) retained; 8 references exist.
- **prompt-governance** — registry YAML schema + eval-type table retained; golden-dataset minimums (20/100+) intact.
- **security-guidance** — `echo '{}' | python3 hooks/security_reminder_hook.py` exits 0 (clean input); pattern table has all 12 rows; attribution block in plugin.json present.
- **statistical-analyst** — `python3 scripts/hypothesis_tester.py --test ztest --control-n 5000 --control-x 250 --treatment-n 5000 --treatment-x 310` exits 0 reporting +1.2pp (verified this audit); effect-size tables intact.
- **terraform-patterns** — `python3 scripts/tf_module_analyzer.py --help` and `tf_security_scanner.py --help` exit 0; review checklist (state, providers, security) retained.
- **workflow-builder** — `python3 scripts/validate_workflow.py --sample` and `workflow_intake.py --help` exit 0; hard-rules list (pure-literal meta, no Date.now, thunks) retained.
- **write-a-skill** — Matt's 3-phase flow + description requirements verbatim; `python3 scripts/skill_review_checklist_runner.py engineering/write-a-skill/skills/write-a-skill` exits 0.
## Agents
| Agent | Verdict | Issue |
|---|---|---|
| agenthub/hub-coordinator | KEEP | Strong: scoped tools allowlist/denylist, hard rules, re-spawn policy |
| autoresearch/experiment-runner | OPTIMIZE | No YAML frontmatter at all (no name/description/tools) — B1 fail; body is good |
| caveman/cs-caveman-mode | KEEP | — |
| claude-coach/cs-claude-coach | KEEP | — |
| grill-me/cs-grill-master | KEEP | Distinct voice + forcing-question pattern |
| grill-with-docs/cs-grill-with-docs | KEEP | — |
| handoff/cs-handoff-author | KEEP | Hard refusals make persona behavioral, not adjectival |
| karpathy-coder/karpathy-reviewer | KEEP | Model exemplar: tool allow/deny lists, exact workflow, report shape |
| llm-wiki/wiki-ingestor | KEEP | — |
| llm-wiki/wiki-librarian | KEEP | — |
| llm-wiki/wiki-linter | KEEP | — |
| universal-scraping-architect/cs-scraping-architect | REWRITE | 6-line placeholder: no tools, no triggers, no workflow — B1/B2/B3 fail |
| workflow-builder/cs-workflow-architect | KEEP | — |
| write-a-skill/cs-skill-author | KEEP | — |
## Commands
| Command | Verdict | Issue |
|---|---|---|
| caveman/cs-caveman | KEEP | Enforces persistence + auto-clarity; wires 3 scripts |
| claude-coach/cs-claude-coach | KEEP | Handles $ARGUMENTS, fixed activation sequence |
| grill-me/cs-grill-me | KEEP | — |
| grill-with-docs/cs-grill-with-docs | KEEP | — |
| handoff/cs-handoff | KEEP | — |
| karpathy-coder/karpathy-check | KEEP | Orchestrates 2 scripts + sub-agent — earns its slot |
| llm-wiki/wiki-init | KEEP | — |
| llm-wiki/wiki-ingest | KEEP | — |
| llm-wiki/wiki-query | KEEP | — |
| llm-wiki/wiki-lint | KEEP | — |
| llm-wiki/wiki-log | KEEP | — |
| universal-scraping-architect/cs-scrape | CUT-OR-MERGE | 6-line placeholder; a bare prompt does strictly more — C3 fail; no $ARGUMENTS handling |
| workflow-builder/cs-workflow-build | KEEP | — |
| write-a-skill/cs-write-a-skill | KEEP | — |
Note: agenthub's 7 `/hub:*` and autoresearch's 5 `/ar:*` surfaces ship as command-style sub-skills (frontmatter `command:` key) rather than `commands/*.md` files — all are substantive and wired; counted under their parent skill verdicts.
## Plugin manifests
1. **engineering-advanced-skills (engineering/.claude-plugin/plugin.json + marketplace.json:227)** — description claims skills not shipped by the manifest: llm-cost-optimizer, prompt-governance, behuman, code-tour, demo-video, data-quality-auditor, statistical-analyst, llm-wiki live in standalone plugin folders, while `"skills": ["./skills"]` only packages `engineering/skills/` (40 dirs). E2 fail.
2. **Triple count mismatch** — bundle index SKILL.md "25 skills" vs plugin description "40" vs 39 actual skills + 1 index dir.
3. **Five orphan plugins** — behuman, claude-coach, grill-with-docs, llm-cost-optimizer, prompt-governance have valid `.claude-plugin/plugin.json` but no marketplace.json entry (handoff is registered as `handoff-engineering`; these five aren't registered at all).
4. **Dual-published duplicates** — slo-architect, chaos-engineering, kubernetes-operator, feature-flags-architect exist byte-identical in two paths; only the standalone copies are marketplace-registered, but the bundle copies also ship via engineering-advanced-skills → users installing both get duplicates with identical trigger descriptions.
5. **Version coherence** — workflow-builder plugin.json is `1.0.0` while every sibling is `2.9.0` (marketplace itself is at v2.10.x per root CLAUDE.md — domain-wide version lag is cosmetic but uniform).
audit/newgen-2026-06/marketing.md
+246
View File
@@ -0,0 +1,246 @@
# Domain audit: marketing-skill/ + marketing/ — new-gen model optimization
Audited: 2026-06-10 · Skills: 49 (47 in marketing-skill/skills + video-content-strategist + marketing/landing) · Agents: 6 · Commands: 4 · Plugins: 5
## Scorecard
| Skill | Verdict | Top issue |
|---|---|---|
| ab-test-setup | OPTIMIZE | Orphan script `sample_size_calculator.py` never named in SKILL.md |
| ad-creative | OPTIMIZE | Stale Meta ">20% image text = reduced distribution" rule (retired 2021) presented as current |
| aeo | KEEP | — |
| ai-seo | CUT-OR-MERGE | Near-total overlap with `aeo` (same goal, no tooling); two skills own one lane |
| analytics-tracking | OPTIMIZE | GA4 "Conversions" terminology (renamed Key events, Mar 2024); script wiring vague |
| app-store-optimization | KEEP | — |
| brand-guidelines | CUT-OR-MERGE | 93-line generic checklist a frontier model already knows; hardcoded Anthropic identity |
| campaign-analytics | KEEP | — |
| churn-prevention | KEEP | — |
| cold-email | KEEP | — |
| competitor-alternatives | OPTIMIZE | Orphan script `comparison_matrix_builder.py` |
| content-creator | CUT-OR-MERGE | Deprecated redirect still shipping 4 refs + 1 asset + agent + zip |
| content-humanizer | OPTIMIZE | 2/6 on repo checklist (worst in domain); AI-tell list itself aging ("delve" era) |
| content-production | OPTIMIZE | 3 of 4 scripts orphaned in SKILL.md |
| content-strategy | OPTIMIZE | Hollow core ("→ see references"); orphan `topic_cluster_mapper.py` |
| copy-editing | OPTIMIZE | Both scripts (`ai_content_detector`, `readability_scorer`) orphaned |
| copywriting | OPTIMIZE | Orphan `headline_scorer.py` |
| email-sequence | OPTIMIZE | Phantom `../../tools/REGISTRY.md` + 5 integration guide links; none exist |
| form-cro | OPTIMIZE | Hollow core; orphan `form_field_analyzer.py` |
| free-tool-strategy | KEEP | — |
| launch-strategy | OPTIMIZE | 74-line shell; orphan `launch_readiness_scorer.py` |
| marketing-context | OPTIMIZE | Writes `.agents/marketing-context.md` while siblings read 2 other paths |
| marketing-demand-acquisition | OPTIMIZE | `updated: 2025-01`, "q1-2025" examples; persona-narrow (Series A+ EU/US) |
| marketing-ideas | KEEP | — |
| marketing-ops | OPTIMIZE | Router missing 11 of 47 skills (incl. aeo, webinar, ASO, x-twitter) |
| marketing-psychology | KEEP | — |
| marketing-skills | CUT-OR-MERGE | Index-as-skill: stale counts (42/7/27 vs actual 47/8/58+), all example paths broken |
| marketing-strategy-pmm | KEEP | — |
| onboarding-cro | OPTIMIZE | Orphan `activation_funnel_analyzer.py` |
| page-cro | OPTIMIZE | Orphan `conversion_audit.py` |
| paid-ads | OPTIMIZE | Phantom tools/REGISTRY.md; 2 orphan scripts; "2024Q1/Mar24" naming examples |
| paywall-upgrade-cro | KEEP | — |
| popup-cro | OPTIMIZE | Hollow core ("→ see references"); zero tooling |
| pricing-strategy | KEEP | — |
| programmatic-seo | OPTIMIZE | Orphan `url_pattern_generator.py` |
| prompt-engineer-toolkit | REWRITE | All 3 references are stubs (328 B / 675 B / 1.5 KB); body ≠ marketing description |
| referral-program | KEEP | — |
| schema-markup | KEEP | — |
| seo-audit | OPTIMIZE | Hollow core; both scripts orphaned |
| signup-flow-cro | OPTIMIZE | Orphan `funnel_drop_analyzer.py` |
| site-architecture | KEEP | — |
| social-content | KEEP | — |
| social-media-analyzer | KEEP | — |
| social-media-manager | OPTIMIZE | Orphan `social_calendar_generator.py` |
| webinar-marketing | OPTIMIZE | `webinar_funnel_scorer.py` has no argparse — `--help` crashes (only D1 fail in domain) |
| x-twitter-growth | KEEP | — |
| youtube-full | KEEP | — |
| video-content-strategist | KEEP | — |
| landing (marketing/) | KEEP | — |
Verdict counts: **KEEP 20 · OPTIMIZE 24 · REWRITE 1 · CUT-OR-MERGE 4**
## Domain-level findings
1. **24 orphan scripts (systemic A3 failure).** Scripts exist and pass `--help` (583/593 repo sweep) but are never named in their own SKILL.md, so a model loading the skill never knows they exist: ab-test-setup, cold-email, competitor-alternatives, content-production (×3), content-strategy, copy-editing (×2), copywriting, email-sequence, form-cro, launch-strategy, marketing-context, marketing-ops, onboarding-cro, page-cro, paid-ads (×2), programmatic-seo, seo-audit (×2), signup-flow-cro, social-media-manager. One repo-wide wiring PR fixes ~half the OPTIMIZE verdicts.
2. **Context-file path schism (foundational pattern silently no-ops).** 19 skills check `.claude/product-marketing-context.md`, 16 check `marketing-context.md`, and the `marketing-context` skill that *creates* the file writes `.agents/marketing-context.md`. Whatever path the user's file is at, half the domain won't find it. Pick one canonical path.
3. **Count drift in 4 places.** marketing-skills SKILL.md: "42 skills / 7 pods / 27 tools"; marketplace.json: "44 skills / 7 pods"; plugin.json + CLAUDE.md: "45 skills / 8 pods"; actual: 47 skill dirs (incl. 1 deprecated redirect and 1 index). All example invocation paths in marketing-skills/SKILL.md omit the `skills/` segment and are broken.
4. **Duplicate AI-search lane.** `aeo` (v2.7.3 port, 3 working scripts, calibrated industry thresholds) and `ai-seo` (older, prose-only) both own "get cited by ChatGPT/Perplexity." The router routes AI-search queries to `ai-seo` and doesn't know `aeo` exists. Merge ai-seo's bot-access/robots.txt + content-pattern material into aeo references.
5. **Router drift.** marketing-ops claims to be the central router but its matrix omits 11 skills: aeo, app-store-optimization, brand-guidelines, marketing-demand-acquisition, marketing-strategy-pmm, prompt-engineer-toolkit, social-media-analyzer, webinar-marketing, x-twitter-growth, youtube-full (+ the marketing-skills index).
6. **Plugin hygiene.** marketing-skill/ root ships 5 .zip archives (~120 KB) and 4 internal planning docs (MARKETING-AUDIT-REPORT.md, -EXECUTION-PLAN.md, -EXPANSION-PLAN.md, marketing_skills_roadmap.md — 1,194 lines) inside the public plugin folder.
7. **Phantom references.** email-sequence and paid-ads link `../../tools/REGISTRY.md` and 5 `../../tools/integrations/*.md` guides; no `tools/` directory exists anywhere in marketing-skill/.
8. **Freshness is better than feared, with 3 specific stales:** Meta's 20%-image-text rule (retired 2021) in ad-creative SKILL + reference; GA4 "Conversions" (renamed "Key events" March 2024) in analytics-tracking; 2024/2025 dating in paid-ads naming examples and marketing-demand-acquisition metadata. x-twitter-growth explicitly labels its algorithm table "2025-2026" — the right pattern.
9. **New-gen model lens: the domain splits cleanly.** Skills that pair calibrated thresholds with deterministic scorers (aeo, campaign-analytics, ASO, churn-prevention, webinar funnel math, landing's validator gate) earn their context. ~8 skills are persona-prompt shells ("Core Principles → see references") whose body adds nothing a frontier model lacks — the value is locked in references the SKILL.md barely indexes.
10. **video-content-strategist plugin not registered.** It has `.claude-plugin/plugin.json` but no entry in root marketplace.json (E3).
## Per-skill findings
### ab-test-setup — OPTIMIZE
Issues: (1) `scripts/sample_size_calculator.py` orphaned — SKILL.md points users to external web calculators instead of its own tool; (2) sample-size quick table good but unverified against the script's output.
Verify: `python3 scripts/sample_size_calculator.py --help` exits 0; SKILL.md contains the literal string `sample_size_calculator.py` with an invocation; script output for baseline 5%/MDE 20% matches the table's ~7k/variant.
### ad-creative — OPTIMIZE
Issues: (1) "Image text <20%" Meta rule stated in SKILL.md spec table and references/platform-specs.md as causing "reduced distribution" — Meta retired the enforcement in 2021; (2) `ad_copy_validator.py` wired but invocation lacks args/format.
Verify: `grep -c "20%" references/platform-specs.md` returns 0 (or the claim is rewritten as historical); `python3 scripts/ad_copy_validator.py --help` exits 0; SKILL.md shows an exact CLI line with input format.
### ai-seo — CUT-OR-MERGE
Issues: (1) duplicates `aeo`'s mission with zero tooling (0 scripts vs aeo's 3); (2) router sends AI-search traffic here, starving aeo; (3) unique value (robots.txt bot matrix, 6 content patterns, GSC AI Overviews monitoring) belongs in aeo/references.
Verify: after merge, `marketing-ops/SKILL.md` routes "AI search/AEO/GEO" triggers to `aeo`; aeo references contain the bot-access table; `ai-seo/` directory removed or reduced to a redirect stub ≤ 30 lines.
### analytics-tracking — OPTIMIZE
Issues: (1) "GA4 → Admin → Conversions" + "Max 30 conversion events" uses pre-2024 terminology (now Key events); (2) `tracking_plan_generator.py` only mentioned in passing in the artifacts table — no CLI invocation or output contract.
Verify: SKILL.md says "Key events"; SKILL.md contains `python3 scripts/tracking_plan_generator.py` with args; `python3 scripts/tracking_plan_generator.py --json` emits parseable JSON.
### brand-guidelines — CUT-OR-MERGE
Issues: (1) 93 lines of generic audit checklist any frontier model reproduces unprompted (A5 fail); (2) "Anthropic Brand Identity" section hardcodes one company's identity into a generic skill; (3) no scripts, 1 reference; (4) overlaps marketing-context §10-11 (Brand Voice + Style Guide).
Verify: brand dimensions folded into marketing-context template (§10/§11 expanded); references/brand-identity-and-framework.md content preserved or moved; routing matrix no longer lists it OR skill rewritten with a deterministic brand-audit scorer.
### competitor-alternatives — OPTIMIZE
Issues: (1) `comparison_matrix_builder.py` orphaned; (2) otherwise strong 4-format framework.
Verify: SKILL.md names `comparison_matrix_builder.py` with exact CLI + consuming step; script `--help` exits 0.
### content-creator — CUT-OR-MERGE
Issues: (1) deprecated redirect skill still ships 4 references + 1 asset that the redirect never uses (A7); (2) `cs-content-creator` agent and `personas/content-strategist` still target it; (3) `content-creator.zip` lingers in plugin root; (4) routing duplicate of one row in marketing-ops.
Verify: references/ + assets/ removed (≤ 1 file redirect remains) or skill deleted with router rows updated; `grep -r "content-creator" agents/` returns no skill-target hits; zip deleted.
### content-humanizer — OPTIMIZE
Issues: (1) 2/6 on repo's own checklist (261 lines, time-sensitive content flags); (2) the AI-tell vocabulary ("delve", "landscape", em-dash) is itself a 2023-24 snapshot — new-gen models have different tells, list needs dating + refresh cadence; (3) "HubSpot published... in 2023" dated example; (4) `humanizer_scorer.py` wired only in artifacts table, no CLI contract.
Verify: `skill_review_checklist_runner.py content-humanizer` ≥ 4/6; SKILL.md shows `python3 scripts/humanizer_scorer.py <file>` with score interpretation thresholds; references/ai-tells-checklist.md carries a "last validated" date.
### content-production — OPTIMIZE
Issues: (1) `brand_voice_analyzer.py`, `content_quality_gates.py`, `seo_optimizer.py` all orphaned — only `content_scorer.py` is named; (2) marketing-skills index advertises these very scripts while the owning skill doesn't.
Verify: all 4 scripts named in SKILL.md with exact CLI; `python3 scripts/content_scorer.py --json` emits JSON with a 0-100 score; Mode 3 consumes brand_voice_analyzer + content_quality_gates outputs by name.
### content-strategy — OPTIMIZE
Issues: (1) core knowledge deferred ("Searchable vs Shareable → see references") leaving a 127-line shell; (2) `topic_cluster_mapper.py` orphaned.
Verify: SKILL.md names `topic_cluster_mapper.py` with invocation; the searchable-vs-shareable decision rule (not just pointer) appears inline; script `--help` exits 0.
### copy-editing — OPTIMIZE
Issues: (1) both scripts (`ai_content_detector.py`, `readability_scorer.py`) orphaned; (2) Seven Sweeps framework is genuinely good — wiring is the only gap.
Verify: each sweep that has a matching script names it (Sweep on AI patterns → ai_content_detector; clarity → readability_scorer); both `--help` exit 0; outputs consumed in the sweep workflow text.
### copywriting — OPTIMIZE
Issues: (1) `headline_scorer.py` orphaned; (2) headline formula section never points at its own scorer.
Verify: SKILL.md "Above the Fold" section invokes `python3 scripts/headline_scorer.py "<headline>"`; script exits 0 with a numeric score.
### email-sequence — OPTIMIZE
Issues: (1) phantom `../../tools/REGISTRY.md` + 5 `tools/integrations/*.md` links — directory doesn't exist; (2) `sequence_analyzer.py` orphaned; (3) core principles deferred to one reference, 135-line shell.
Verify: `grep -c "tools/REGISTRY" SKILL.md` returns 0; `sequence_analyzer.py` named with CLI; all relative links in SKILL.md resolve (`find` check).
### form-cro — OPTIMIZE
Issues: (1) `form_field_analyzer.py` orphaned; (2) hollow core ("Core Principles → see references").
Verify: script named with CLI and consumed in the audit output format; field-count/friction thresholds inline in SKILL.md (not only in playbook).
### launch-strategy — OPTIMIZE
Issues: (1) 74 lines — everything substantive deferred to one reference; (2) `launch_readiness_scorer.py` orphaned; (3) thinnest non-deprecated skill in domain.
Verify: SKILL.md ≥ inline ORB definition + phase model summary; `launch_readiness_scorer.py` named with CLI; `--help` exits 0.
### marketing-context — OPTIMIZE
Issues: (1) writes `.agents/marketing-context.md` while 19 sibling skills read `.claude/product-marketing-context.md` and 16 read `marketing-context.md` — the foundation file is invisible to half its consumers; (2) `context_validator.py` orphaned.
Verify: one canonical path declared and used by this skill's output instruction; `grep -rl "product-marketing-context" skills/*/SKILL.md` and `grep -rl "marketing-context.md"` agree with that path; `python3 scripts/context_validator.py --json` emits completeness score 0-100 and is named in SKILL.md.
### marketing-demand-acquisition — OPTIMIZE
Issues: (1) `metadata.updated: 2025-01`, "q1-2025-linkedin-enterprise" examples; (2) description hard-scopes to "Series A+ scaling internationally EU/US/Canada hybrid PLG/Sales-Led" — over-narrow trigger for a general demand-gen skill; (3) HubSpot-specific workflows presented as the default stack.
Verify: dates refreshed or genericized; description triggers cover demand-gen broadly with the persona as a default profile not a gate; campaign workflow validation step still names the UTM-in-CRM check.
### marketing-ops — OPTIMIZE
Issues: (1) routing matrix omits 11 skills (aeo, ASO, webinar-marketing, x-twitter-growth, social-media-analyzer, marketing-strategy-pmm, marketing-demand-acquisition, brand-guidelines, prompt-engineer-toolkit, youtube-full, video-content-strategist); (2) AI-search row routes to ai-seo only; (3) `campaign_tracker.py` orphaned.
Verify: `for d in skills/*/; do grep -q "$(basename $d)" marketing-ops/SKILL.md || echo MISS; done` prints nothing (minus deliberate exclusions); aeo present in SEO pod rows; `campaign_tracker.py` named with CLI.
### marketing-skills — CUT-OR-MERGE
Issues: (1) it's a README/index in SKILL.md clothing — no workflow, no trigger utility; (2) stale counts (42 skills/7 pods/27 tools vs actual 47/8/58+); (3) every example path broken (omits `skills/` segment: `marketing-skill/content-production/scripts/...`); (4) references 6 scripts that live in other skills (phantom `scripts/*.py` paths); (5) duplicates marketing-ops (router) and README.md (index).
Verify: file demoted to README.md (or deleted) and removed from skill counts; if kept as SKILL.md, all paths resolve (`grep -oE "marketing-skill[^ )]*" | xargs -I{} test -e {}`) and counts match `ls skills/ | wc -l`.
### onboarding-cro — OPTIMIZE
Issues: (1) `activation_funnel_analyzer.py` orphaned; (2) no references dir — all knowledge inline (acceptable) but no verification loop.
Verify: script named with CLI; `--help` exits 0; output (drop-off by step) consumed in the audit output format section.
### page-cro — OPTIMIZE
Issues: (1) `conversion_audit.py` orphaned; (2) framework is solid but ends with no machine-checkable gate (A4).
Verify: SKILL.md invokes `python3 scripts/conversion_audit.py` and the audit report format references its score; `--help` exits 0.
### paid-ads — OPTIMIZE
Issues: (1) phantom `../../tools/REGISTRY.md` reference; (2) `ad_health_scorer.py` + `roas_calculator.py` both orphaned; (3) naming-convention examples dated "2024Q1"/"Mar24"; (4) 5 refs exist but only 1 linked from SKILL.md.
Verify: zero phantom links; both scripts named with CLI; `python3 scripts/roas_calculator.py --help` exits 0; example campaign names use current-year placeholders or `{YYYY}` tokens.
### popup-cro — OPTIMIZE
Issues: (1) hollow core ("Core Principles → see references/popup-cro-playbook.md"); (2) zero tooling; (3) experiment lists are the kind of generic ideation a frontier model produces unaided (A5 risk).
Verify: trigger-timing and frequency-cap thresholds (the calibrated part of the playbook) surfaced inline; SKILL.md ≤ 250 lines with decision rules, not idea lists.
### programmatic-seo — OPTIMIZE
Issues: (1) `url_pattern_generator.py` orphaned; (2) otherwise strong (data-defensibility hierarchy, penalty avoidance).
Verify: script named with CLI in the page-generation workflow; `--help` exits 0.
### prompt-engineer-toolkit — REWRITE
Issues: (1) references are stubs: evaluation-rubric.md 328 B, technique-guide.md 675 B, prompt-templates.md 1.5 KB — no citations, no marketing templates despite the description promising "prompt templates for marketing use cases (ad copy, email campaigns, social media)" (A1/A7 fail); (2) body is generic LLM-feature governance, not marketing — arguably belongs in engineering/; (3) scripts (`prompt_tester.py`, `prompt_versioner.py`) are real and wired — keep them.
Verify: each reference ≥ 3 KB with ≥ 5 cited sources OR skill relocated to engineering/ with description rewritten to match the body; prompt-templates.md contains ≥ 5 concrete marketing prompt templates if it stays in marketing; both scripts still pass `--help`.
### seo-audit — OPTIMIZE
Issues: (1) both scripts (`seo_checker.py`, `seo_health_scorer.py`) orphaned; (2) audit framework wholly deferred ("→ See references/seo-audit-reference.md"); (3) 4 references are good (CWV thresholds, E-E-A-T) but SKILL.md gives the model no decision rules inline.
Verify: both scripts named with CLI and consumed in the report structure; CWV pass/fail thresholds (LCP/INP/CLS numbers) inline in SKILL.md; `--help` exits 0 on both.
### signup-flow-cro — OPTIMIZE
Issues: (1) `funnel_drop_analyzer.py` orphaned.
Verify: script named with CLI; `--help` exits 0; output consumed in audit format.
### social-media-manager — OPTIMIZE
Issues: (1) `social_calendar_generator.py` orphaned; (2) overlaps social-content's platform tables (duplicate cadence specs that can drift independently).
Verify: script named with CLI; cadence table either deduplicated with social-content or marked as the canonical copy; `--help` exits 0.
### webinar-marketing — OPTIMIZE
Issues: (1) **confirmed known issue:** `scripts/webinar_funnel_scorer.py` has no argparse — `python3 ... --help` raises FileNotFoundError treating `--help` as a JSON filename (only D1 failure in marketing scope; it is the skill's only script, no affected siblings); (2) SKILL.md itself is among the best in the domain (backward funnel math, benchmark-tagged outputs) — fix is surgical.
Verify: `python3 scripts/webinar_funnel_scorer.py --help` exits 0 and prints usage; no-arg run still executes embedded sample and prints `WEBINAR FUNNEL SCORE: \d+/100` plus a JSON block; `echo '{}' | python3 scripts/webinar_funnel_scorer.py -` doesn't crash.
## KEEP-verdict verification criteria
- **aeo** — all 3 scripts pass `--help`; `aeo_audit.py --input <md> --output json` emits composite 0-100 + 4 dimension keys; industry table thresholds (healthcare/finance/legal ≥ 85) unchanged.
- **app-store-optimization** — all 8 scripts pass `--help`; `metadata_optimizer.py --platform ios --title "<31 chars>"` flags the over-limit title; char-limit table matches Apple 30/30/100 + Play 50/80.
- **campaign-analytics** — 3 scripts run on `assets/sample_campaign_data.json` with `--format json` exit 0; attribution output includes all 5 models; funnel analyzer names a bottleneck stage.
- **churn-prevention** — `churn_impact_calculator.py` runs no-arg with sample, exits 0; benchmark table retains save-rate ≥ 10-15% / recovery 25-35% calibration.
- **cold-email** — 3 references resolve; deliverability section still names SPF/DKIM/DMARC + warmup ramp; orphan `email_sequence_analyzer.py` gets wired (carryover from A3 sweep).
- **free-tool-strategy** — `tool_idea_scorer`-class script passes `--help` and is named in Mode 1; 6-factor evaluation framework present.
- **marketing-ideas** — references/ideas-by-category.md contains all 139 numbered ideas; SKILL.md stage-mapping numbers (#79, #81...) resolve to entries in the reference.
- **marketing-psychology** — references/mental-models-catalog.md contains ≥ 70 models; the 6-category count table matches the catalog's actual counts.
- **marketing-strategy-pmm** — 4 references resolve; April Dunford positioning workflow + ICP validation checklist intact; no scripts claimed (none promised).
- **paywall-upgrade-cro** — self-contained; description's distinct-from-pricing-page boundary preserved; router row intact.
- **pricing-strategy** — `pricing_modeler.py` passes `--help` and stays named in SKILL.md; three-axes model + Van Westendorp trigger words remain in description.
- **referral-program** — incentive-structure script passes `--help` and is named; LTV-ceiling logic for incentive sizing intact.
- **schema-markup** — `schema_validator.py` named in Mode 1 step 1 and passes `--help`; schema selection table covers FAQ/HowTo/Article/Product/Organization/Person.
- **site-architecture** — `sitemap_analyzer.py` named in Mode 1 and passes `--help`; subfolder-over-subdomain rule intact.
- **social-content** — references/platforms.md + post-templates.md resolve; cadence table consistent with social-media-manager's (post-dedup).
- **social-media-analyzer** — both scripts pass `--help`; engagement-rate formula (engagements/reach×100) and validation gates (ER < 100%) intact.
- **x-twitter-growth** — all 5 scripts pass `--help` and stay named; algorithm-signal table keeps an explicit date label ("2025-2026" or refreshed); cadence-by-account-size table intact.
- **youtube-full** — BYOK note + OSS fallback table intact; fix 3 cross-reference paths (`marketing-skill/skills/video-content-strategist` → actual location); endpoint table matches credit-cost table.
- **video-content-strategist** — niche/positioning framework + 90-day plan intact; plugin gets registered in marketplace.json (currently missing).
- **landing (marketing/)** — 3 scripts pass `--help`; `html_validator.py --file <out>.html` checks all 8 listed gates; GSAP CDN pins resolve; SKILL.md keeps gsap.set()-before-timeline FOUC rule.
## Agents
| Agent | Verdict | Notes |
|---|---|---|
| agents/marketing/cs-aeo.md | KEEP | B1-B3 pass; differentiated voice (refuses fake authority, AEO≠SEO routing); trigger phrasing present. |
| agents/marketing/cs-webinar-marketer.md | KEEP | Differentiated (refuses vanity metrics; fixes the broken stage); trigger phrasing present. |
| marketing/landing/agents/cs-landing.md | KEEP | Forcing-intake persona with concrete refusals; wired to skill + scripts. |
| agents/marketing/cs-content-creator.md | CUT-OR-MERGE | Targets the **deprecated** content-creator skill via a wrong path (`marketing-skill/content-creator`); no trigger phrasing (B1 fail); 3 paragraphs of swappable boilerplate (B2/B3 fail). Retarget to content-production or delete. |
| agents/marketing/cs-demand-gen-specialist.md | REWRITE | No trigger phrasing; wrong skill path (`marketing-skill/marketing-demand-acquisition`, missing `skills/`); body is the same boilerplate template as cs-content-creator — swappable (B2 fail). |
| agents/personas/content-strategist.md | OPTIMIZE | Skills list includes deprecated `content-creator`; otherwise differentiated persona frontmatter. |
## Commands
| Command | Verdict | Notes |
|---|---|---|
| commands/cs-aeo.md | KEEP | C1-C3 pass; orchestrates 3 scripts with modes; explicit when-NOT-to-run; distinct from /cs:seo-audit documented. |
| commands/cs-webinar.md | KEEP | C1-C3 pass; plan/rescue/evergreen modes; gates on funnel-math feasibility. |
| marketing/landing/commands/cs-landing.md | KEEP | C1-C3 pass; 4-question forcing intake + validator gate; routes away to landing-page-generator when conversion-optimized output needed. |
| commands/seo-auditor.md | KEEP | Repo-docs SEO utility (not marketing-skill plugin surface); 7-phase pipeline with report-only flag; does real orchestration. |
Gap: no `/cs:*` commands exist for the other 45 marketing skills — the domain relies on skill triggers alone. Acceptable, but the marketing-skills index promises slash-command-grade entry points it doesn't have.
## Plugin manifests
| Manifest | Verdict | Notes |
|---|---|---|
| marketing-skill/.claude-plugin/plugin.json | OPTIMIZE | Schema valid (`"skills": ["./skills"]` ✓). E2 fail: description says "45 skills across 8 pods" — actual 47 skill dirs (incl. deprecated redirect + index); marketplace.json entry says 44/7; marketing-skills SKILL.md says 42/7. Three counts, four places. |
| marketing-skill/skills/aeo/.claude-plugin/plugin.json | KEEP | Schema valid; `source` extension block documented; description matches contents. |
| marketing-skill/skills/youtube-full/.claude-plugin/plugin.json | KEEP | Schema valid; attribution block present; BYOK disclosed (ClawHub rule 3 satisfied — free tier + OSS fallbacks documented). |
| marketing-skill/video-content-strategist/.claude-plugin/plugin.json | OPTIMIZE | Schema valid but **not registered in root marketplace.json** (E3 fail) — plugin is undiscoverable. |
| marketing/landing/.claude-plugin/plugin.json | OPTIMIZE | Schema valid; `source` block present. E2 drift: marketplace.json description for `landing` describes the *other* landing skill ("4 design styles, brand palette validation" = product-team TSX generator language), not this GSAP/HTML one. |
Hygiene: marketing-skill/ plugin root ships 5 .zip archives and 4 internal planning markdown docs (1,194 lines) that don't belong in a distributed plugin.
audit/newgen-2026-06/product-pm.md
+150
View File
@@ -0,0 +1,150 @@
# Domain audit: product-team/ + project-management/ — new-gen model optimization
Audited: 2026-06-10 · Skills: 26 (17 product-team + 9 project-management) · Agents: 6 · Commands: 11 · Plugins: 6 (+1 .mcp.json)
## Scorecard
| Skill | Verdict | Top issue |
|---|---|---|
| product-team/skills/product-skills (router) | CUT-OR-MERGE | 61-line index with broken `/read` paths and stale counts; adds no orchestration |
| product-team/skills/product-manager-toolkit | OPTIMIZE | Two contradictory RICE CSV schemas in one file; ~120 lines of generic PM advice |
| product-team/agile-product-owner | KEEP | — |
| product-team/skills/product-strategist | OPTIMIZE | Generator emits canned OKR prose a frontier model writes better; keep the alignment scorer |
| product-team/skills/ux-researcher-designer | KEEP | — |
| product-team/skills/ui-design-system | KEEP | — |
| product-team/skills/competitive-teardown | OPTIMIZE | Its only script (competitive_matrix_builder.py) is never referenced in SKILL.md |
| product-team/skills/landing-page-generator | OPTIMIZE | Own script + all 4 references orphaned; overlaps marketing/landing |
| product-team/skills/saas-scaffolder | OPTIMIZE | project_bootstrapper.py orphaned; stack pins aging (Next.js 14, NextAuth v4) |
| product-team/skills/product-analytics | KEEP | — |
| product-team/skills/experiment-designer | KEEP | — |
| product-team/skills/product-discovery | KEEP | — |
| product-team/skills/roadmap-communicator | OPTIMIZE | Mostly framework-explainer prose; value is the changelog tool + templates |
| product-team/skills/spec-to-repo | KEEP | — |
| product-team/code-to-prd | OPTIMIZE | Malformed frontmatter (duplicate `Name:`/`name:` + non-standard keys) |
| product-team/research-summarizer | CUT-OR-MERGE | Duplicates research/ domain (litreview/dossier); phantom `/research:*` commands |
| product-team/apple-hig-expert | REWRITE | Known 3/6; persona filler, no exact CLI, citation-free references with dubious claims |
| project-management/skills/pm-skills (router) | CUT-OR-MERGE | Thin index; broken paths; says 6 skills, domain has 9 |
| project-management/skills/senior-pm | OPTIMIZE | Strong quant core buried in ~150 lines of governance boilerplate; invented KPI targets |
| project-management/skills/scrum-master | KEEP | — |
| project-management/skills/jira-expert | REWRITE | Fabricated MCP syntax (`mcp jira create_project --flags`) naming tools the bundled server doesn't have |
| project-management/skills/confluence-expert | REWRITE | Fabricated MCP tool names + phantom MACROS.md/PERMISSIONS.md + legacy wiki markup for Cloud |
| project-management/skills/atlassian-admin | OPTIMIZE | Factual errors (DELETE /rest/api/3/user ≠ deactivate; "7 years for GDPR"); script orphaned |
| project-management/skills/atlassian-templates | REWRITE | Claims "exact parameter names expected by the Atlassian MCP server" for tools that don't exist |
| project-management/skills/meeting-analyzer | KEEP | — |
| project-management/skills/team-communications | KEEP | — |
**Totals:** KEEP 10 · OPTIMIZE 9 · REWRITE 4 · CUT-OR-MERGE 3
## Domain-level findings
1. **MCP wiring is fiction in 3 of the 4 Atlassian skills (highest-severity finding).** The bundled `.mcp.json` correctly points to the Atlassian Remote MCP (`https://mcp.atlassian.com/v1/sse`, SSE), whose real tools are camelCase: `createJiraIssue`, `editJiraIssue`, `searchJiraIssuesUsingJql`, `transitionJiraIssue`, `createConfluencePage`, `updateConfluencePage`, `searchConfluenceUsingCql`, etc. The skills document three different *invented* conventions, none matching: jira-expert uses CLI-flag pseudo-syntax (`mcp jira create_project --name ... --type scrum`), confluence-expert uses snake_case JS calls (`create_space({...})`, `delete_page`, `add_label`), atlassian-templates uses JSON tool blocks (`confluence_create_page`, `jira_update_field_configuration`) while asserting these are "the exact parameter names expected by the Atlassian MCP server." Several referenced capabilities (`create_project`, `create_sprint`, `create_filter`, `create_space`, field-configuration editing) do not exist on the Remote MCP at all. `project-management/CLAUDE.md` adds a fourth convention (`mcp__atlassian__create_issue`, `mcp__atlassian__create_sprint`, `mcp__atlassian__link_issue`). A new-gen model following any of these will emit failing tool calls.
2. **8 orphaned scripts (A3 systemic).** Script dirs exist but SKILL.md never invokes them: competitive-teardown, landing-page-generator, saas-scaffolder, atlassian-admin (1 each); jira-expert, confluence-expert (2 each); atlassian-templates (1). The repo-wide smoke test passes them (D1 fine) but no workflow consumes their output — dead weight per the rubric.
3. **Stale path layout in agents, commands, and routers.** Skills moved under `*/skills/` subdirs but `agents/product/*` reference `../../product-team/product-manager-toolkit/`, commands reference `project-management/scrum-master/SKILL.md`, and both router skills instruct `/read product-team/product-manager-toolkit/SKILL.md`. All resolve to nothing.
4. **Count drift everywhere.** product-skills plugin.json: "13 production-ready product skills" then lists 16 names. product-skills SKILL.md: "10" in description, "8" in body, 13 on disk. pm-skills: 9 in marketplace, 6 in plugin description and SKILL.md, 9 on disk. project-management/CLAUDE.md claims 9 skills but documents only 6 — meeting-analyzer and team-communications (two of the three best skills in the domain) are invisible to it.
5. **PM references are citation-free**, consistent with the repo-wide flag: jql-examples.md (0 sources), team-dynamics-framework.md (0), governance-framework.md (0), template-design-patterns.md (0), retro-formats.md (1). Only senior-pm's prioritization/risk references cite anything.
6. **Generic-knowledge dead weight (A2).** "Handoff Protocols", "Best Practices", "Common Pitfalls" sections across senior-pm, jira-expert, confluence-expert, product-manager-toolkit, roadmap-communicator restate what a frontier model already knows. The skills that skip this (product-analytics, experiment-designer, meeting-analyzer, scrum-master) are the domain's best.
## Per-skill findings
### product-team/skills/product-skills — CUT-OR-MERGE
- Issues: (1) Pure index page — no routing logic, no classifier, nothing a model can execute. (2) Quick Start path `/read product-team/product-manager-toolkit/SKILL.md` is wrong (missing `skills/`). (3) Description says 10 skills, body table lists 8, directory holds 13. (4) Duplicates product-team/CLAUDE.md content.
- Verify: `grep -c "product-team/skills/" product-team/skills/product-skills/SKILL.md` ≥ 1 if retained; otherwise plugin.json `"skills"` array still validates via `python3 scripts/check_plugin_json.py --all` after removal.
### product-team/skills/product-manager-toolkit — OPTIMIZE
- Issues: (1) Two incompatible RICE CSV schemas shown — Quick Start (`feature,reach,impact,confidence,effort` numeric) vs Tools Reference (`name,reach,impact,confidence,effort,description` with `high`/`massive`/`l` categorical values); only one can match the script's parser. (2) ~120 lines of generic PM advice (Best Practices, Pitfalls tables) a frontier model already knows. (3) Verification loop is checklist-only — no machine-checkable gate.
- Verify: `python3 scripts/rice_prioritizer.py sample && python3 scripts/rice_prioritizer.py sample_features.csv --output json` exits 0 and emits JSON with per-feature `rice_score`; the single canonical CSV header in SKILL.md matches `sample_features.csv` byte-for-byte; `python3 scripts/customer_interview_analyzer.py <file> json` exits 0 emitting `pain_points` key.
### product-team/skills/product-strategist — OPTIMIZE
- Issues: (1) Generator output is canned objective prose ("Build viral product features...") — new-gen models write better OKRs unaided; the durable value is the 4-score alignment math (vertical/horizontal/coverage/balance) and thresholds. (2) Sample output hardcodes "Q1 2025" (A6 minor). (3) No verification loop beyond a checklist.
- Verify: `python3 scripts/okr_cascade_generator.py growth --json | python3 -c "import json,sys; d=json.load(sys.stdin); assert d['alignment_scores']['overall']>0"` exits 0; thresholds table (>90/>75/>80/>80) still present in SKILL.md after trimming.
### product-team/skills/competitive-teardown — OPTIMIZE
- Issues: (1) `scripts/competitive_matrix_builder.py` exists but is never mentioned — the 12-dimension rubric is manual-only. (2) Step 4 templates live in `references/analysis-templates.md`; the workflow never says when to load it vs the inline summary (mild progressive-disclosure confusion). (3) No final verification gate after step 6.
- Verify: `python3 scripts/competitive_matrix_builder.py --help` exits 0 AND SKILL.md contains an exact `python3 scripts/competitive_matrix_builder.py` invocation whose output feeds step 3 (scorecard); validation checkpoint at step 2 (pricing + ≥20 reviews + job counts) retained.
### product-team/skills/landing-page-generator — OPTIMIZE
- Issues: (1) `scripts/landing_page_scaffolder.py` (the tool product-team/CLAUDE.md advertises for this skill) is never referenced — only marketing-skill's brand_voice_analyzer is. (2) All 4 reference files (conversion-patterns, copy-frameworks, landing-page-patterns, seo-checklist) unreferenced. (3) Functional overlap with `marketing/landing` (v2.7.0) — needs an explicit distinct-from note (TSX/Next.js vs single-file HTML). (4) SEO "validation step" is honor-system, no executable check.
- Verify: `python3 scripts/landing_page_scaffolder.py --help` exits 0 and SKILL.md shows `--format tsx|html` invocation consumed by the generation workflow; a "distinct from marketing/landing" sentence exists; all 4 reference files cited or deleted.
### product-team/skills/saas-scaffolder — OPTIMIZE
- Issues: (1) `scripts/project_bootstrapper.py` orphaned — phase checklist never calls it. (2) Freshness: NextAuth v4 patterns (`NextAuthOptions`, `getServerSession`) and "Next.js 14+" pins will mislead in 2026 (Auth.js v5 / Next 15 era). (3) Reference Files section asks the model to *generate* CUSTOMIZATION.md/PITFALLS.md/BEST_PRACTICES.md rather than shipping them (A7 — shells).
- Verify: `python3 scripts/project_bootstrapper.py --help` exits 0 and is invoked in Phase 1 of the checklist; Phase 4 webhook idempotency validation retained; stack-version claims dated or generalized.
### product-team/skills/roadmap-communicator — OPTIMIZE
- Issues: (1) ~70% of body is audience-framing advice a frontier model knows (board = outcomes, engineers = dependencies). (2) Only real asset is `changelog_generator.py` + two template references; quality checklist is non-executable. (3) No JSON-mode mention despite the script supporting `--json` (per CLAUDE.md).
- Verify: from a git repo, `python3 scripts/changelog_generator.py --from <tag> --to HEAD --json` exits 0 emitting grouped conventional-commit entries; both `references/roadmap-templates.md` and `references/communication-templates.md` load (exist, non-empty).
### product-team/code-to-prd — OPTIMIZE
- Issues: (1) Frontmatter contains duplicate keys (`Name:` and `name:`) plus non-standard `Tier/Category/Dependencies/Author/Version` at top level — fragile under strict YAML parsers and fails the repo's own description conventions. (2) 507 lines; the README/per-page templates could move to `references/` (progressive disclosure). (3) Otherwise the strongest large skill in the domain (mock-detection signals, enum exhaustiveness, [TBC] uncertainty rule).
- Verify: `python3 -c "import yaml; yaml.safe_load(open('SKILL.md').read().split('---')[1])"` parses with exactly one `name` key; `python3 scripts/codebase_analyzer.py <dir> -o analysis.json && python3 scripts/prd_scaffolder.py analysis.json -o /tmp/prd` exits 0 producing `prd/README.md`.
### product-team/research-summarizer — CUT-OR-MERGE
- Issues: (1) Core capability (structured summarization) is native frontier-model behavior; wrapper value is one regex citation extractor. (2) Direct overlap with `research/litreview`, `research/dossier`, `research/notebooklm` (v2.7.0) — no disambiguation anywhere. (3) Advertises `/research:summarize|compare|cite` slash commands that exist nowhere in `commands/` (phantom). (4) `format_summary.py` emits empty templates — a template printer, not analysis. (5) Installation section references `./scripts/convert.sh` not shipped with the skill.
- Verify: if retained, `grep -r "research:summarize" commands/` returns ≥1 file or the Slash Commands section is removed; a "distinct from research/litreview" note exists; `python3 scripts/extract_citations.py <file> --output json` exits 0 with deduplicated entries.
### product-team/apple-hig-expert — REWRITE
- Issues: (1) Confirmed 3/6 on repo checklist: description has no "Use when" trigger phrases; "You are a Senior Apple Design Lead with decades of experience" is exactly the A2 filler the rubric bans; zero concrete examples (no sample audit, no before/after). (2) A3 fail: "Run the `hig_checker.py` tool" with no invocation — actual CLI is `hig_checker.py {contrast,target,batch}` with subcommand args the SKILL.md never shows. (3) References cite zero sources (no developer.apple.com links) and contain unverifiable claims ("SF Camera" as a public SF variant; "Liquid Glass introduced in late 2025" — WWDC25 was June 2025) — for a freshness-critical Apple skill this is fatal. (4) 90 lines of pointers with the actual expertise missing: no Liquid Glass API names (`glassEffect`, materials hierarchy), no per-platform metric tables in SKILL.md. (5) Scorecard "0-100" output promised with no rubric to compute it.
- Verify: `python3 scripts/hig_checker.py contrast --help && python3 scripts/hig_checker.py batch --help` exit 0 and both invocations appear verbatim in SKILL.md; description matches `Use when` trigger regex of `scripts/audit_skills.py` (skill scores ≥5/6 on `skill_review_checklist_runner.py`); every reference doc cites ≥3 developer.apple.com URLs; at least one worked audit example (input mockup description → scored findings) present.
### project-management/skills/pm-skills — CUT-OR-MERGE
- Issues: (1) Index-only router; `/read project-management/jira-expert/SKILL.md` path wrong (missing `skills/`). (2) Claims 6 skills; domain ships 9 — meeting-analyzer, team-communications invisible. (3) Example tool paths (`senior-pm/scripts/...`) also missing the `skills/` segment.
- Verify: every path in the file resolves (`while read p; do test -e "$p"; done`) or the file is removed and pm-skills plugin.json still passes `check_plugin_json.py --all`.
### project-management/skills/senior-pm — OPTIMIZE
- Issues: (1) ~150 lines of Handoff Protocols / Continuous Improvement / Stakeholder Feedback boilerplate (A2 dead weight). (2) Success-metric targets (">70% risk prediction accuracy", "10% transformational") presented with no source or basis (A5 weakness). (3) Description promises "Monte Carlo simulation" — confirm `risk_matrix_analyzer.py` actually simulates rather than just the formula snippets shown. (4) Quant core (EMV, category weights, three-point estimation, response thresholds >18/12-18/8-12/<8, STOP gates) is genuinely good — keep all of it.
- Verify: all three scripts run against `assets/sample_project_data.json` exiting 0; `project_health_dashboard.py ... --format json` emits composite score + RAG consistent with the >80/60-80/<60 thresholds documented; if Monte Carlo isn't in the scripts, the claim is removed from the description.
### project-management/skills/jira-expert — REWRITE
- Issues: (1) Every "MCP" example uses invented CLI syntax (`mcp jira create_project --name "My Project" --type scrum`) — not MCP, not the bundled server. Real server: `createJiraIssue`, `searchJiraIssuesUsingJql`, `editJiraIssue`, `transitionJiraIssue`; it has NO project/sprint/filter creation. (2) Both scripts (`jql_query_builder.py`, `workflow_validator.py` — which work and produce useful JQL) are never mentioned. (3) `--startDate "2024-06-01"` 2024-ism (A6). (4) JQL operator/function content is generic knowledge a frontier model has; the JQL examples references cite 0 sources. (5) Verdict REWRITE not CUT: the JQL recipes + escalation framework + the two scripts are a salvageable core.
- Verify: every MCP call in SKILL.md names a tool that exists on the Atlassian Remote MCP (assert each appears in the server's tool list; non-existent operations rewritten as REST-API or UI steps); `python3 scripts/jql_query_builder.py "high priority bugs assigned to me"` exits 0 emitting valid JQL and is wired into the JQL workflow; no pre-2026 literal dates.
### project-management/skills/confluence-expert — REWRITE
- Issues: (1) MCP examples (`create_space`, `update_page`, `delete_page`, `get_children`, `add_label`) don't match the real server (`createConfluencePage`, `updateConfluencePage`, `getConfluencePage`, `getConfluencePageDescendants`, `searchConfluenceUsingCql`; no space-creation/delete/label tools). (2) Phantom references: cites `MACROS.md`, `TEMPLATES.md`, `PERMISSIONS.md` — actual files are `macro-cheat-sheet.md`, `templates.md`, `space-architecture-patterns.md`; PERMISSIONS.md doesn't exist at all. (3) Macro sections teach legacy wiki markup (`{info}`, `{section}{column}`) which Confluence Cloud pages (what the MCP writes, storage-format XHTML/ADF) won't accept. (4) Scripts (`space_structure_generator.py`, `content_audit_analyzer.py`) orphaned. (5) Long generic sections (governance, handoffs) a model already knows.
- Verify: every file path cited in SKILL.md exists (`grep -oE '[A-Za-z-]+\.md' SKILL.md` all resolve under references/); every MCP example names a real Remote-MCP tool; macro examples shown in storage format with a note on wiki-markup legacy; both scripts invoked with exact CLI and outputs consumed by Space Creation / KB audit workflows.
### project-management/skills/atlassian-admin — OPTIMIZE
- Issues: (1) Factual errors: `DELETE /rest/api/3/user` deletes (org-admin deactivation is a different endpoint) yet documented as "Deactivate"; "minimum 7 years for SOC 2/GDPR compliance" is invented — GDPR mandates no such retention and pushes minimization. (2) `permission_audit_tool.py` orphaned. (3) Strength: admin.atlassian.com click-paths and REST endpoints are real specificity (A5 pass) — admin operations aren't covered by the Remote MCP, so the "Atlassian MCP Integration" closing section over-promises and should be cut or scoped. (4) References dir exists but is never pointed to from the workflows.
- Verify: `python3 scripts/permission_audit_tool.py --help` exits 0 and an exact invocation appears in the Permission Scheme Design workflow; deactivation step cites the org API (`POST /admin/v1/orgs/{orgId}/directory/users/{accountId}/suspend-access`) or the console path only; retention claim sourced or deleted.
### project-management/skills/atlassian-templates — REWRITE
- Issues: (1) States "All MCP calls below use the exact parameter names expected by the Atlassian MCP server" — then lists `confluence_create_page`, `confluence_update_page`, `confluence_get_page`, `jira_update_field_configuration`, none of which exist (real: `createConfluencePage`/`updateConfluencePage`/`getConfluencePage`; no field-configuration tool at all). False precision is worse than vagueness. (2) Phantom references: `TEMPLATES.md` and `HANDOFFS.md` cited; actual files are `governance-framework.md`, `template-design-patterns.md`. (3) Conflates "Confluence storage format (wiki markup)" — storage format is XHTML, wiki markup is a different legacy syntax; the example is wiki markup and will not round-trip through the Cloud API. (4) `template_scaffolder.py` (which generates storage-format XHTML per CLAUDE.md — the actually-correct artifact) is never mentioned.
- Verify: `python3 scripts/template_scaffolder.py meeting-notes` exits 0 emitting storage-format XHTML and is the documented deployment path; every MCP tool named in SKILL.md exists on the Remote MCP; cited reference filenames resolve on disk; "storage format" vs "wiki markup" used correctly throughout.
## KEEP-verdict verification criteria
- **agile-product-owner**: AC-count-by-points table (1-2→3-4 … 13+→split) and availability-factor table intact; `python3 scripts/user_story_generator.py sprint 30` exits 0 emitting committed ≤85% of capacity; weighted prioritization (40/30/15/15) unchanged.
- **ux-researcher-designer**: `python3 scripts/persona_generator.py json` exits 0 with `confidence` field; sample-size confidence table (5-10 low / 11-30 med / 31+ high) and severity 1-4 table retained.
- **ui-design-system**: `python3 scripts/design_token_generator.py "#0066CC" modern json` exits 0 emitting color/typography/spacing token groups; WCAG AA thresholds (4.5:1 / 3:1) stated in Workflow 1 step 5.
- **product-analytics**: all three subcommands (`retention`, `cohort`, `funnel`) run against documented CSV headers with `--format json` exiting 0; anti-pattern table (6 rows) retained.
- **experiment-designer**: `python3 scripts/sample_size_calculator.py --baseline-rate 0.12 --mde 0.02 --mde-type absolute` exits 0 printing `n_per_group`/`n_total`; If/Then/Because hypothesis gate and peeking warning retained.
- **product-discovery**: `python3 scripts/assumption_mapper.py <csv>` exits 0 emitting prioritized test plan; OST quality gates (≥3 opportunities, ≥2 experiments/opportunity) retained.
- **spec-to-repo**: `python3 scripts/validate_project.py <dir> --format json` exits 0; Phase 4 checklist wired to the script; anti-pattern table (9 rows incl. phantom-imports) retained.
- **scrum-master**: all 3 scripts run on `assets/sample_sprint_data.json` matching `assets/expected_output.json` anchors (velocity avg ≈20.2, CV ≈12.7%, health ≈78.3); <3-sprint refusal gate fires on truncated input.
- **meeting-analyzer**: calibrated thresholds preserved (filler >3/100 words, speaking share >60%, interruption >2:1, 3+ meetings for trends); no script claims introduced without scripts existing.
- **team-communications**: all 4 routing targets (`references/3p-updates.md` etc.) exist and load; ambiguity rule ("ask one clarifying question") retained.
## Agents
6 agents (5 product + 1 PM), all `model: sonnet`, all `tools: [Read, Write, Bash, Grep, Glob]`.
- **B1 partial fail (all 6):** descriptions state capability but no trigger phrasing ("Use when…" / "Use PROACTIVELY") — they read like catalog blurbs.
- **Stale skill paths (all 6):** `skills:` values and body paths (`../../product-team/product-manager-toolkit/`, `../../project-management/senior-pm/scripts/...`) predate the `skills/` subdirectory move; none resolve. cs-project-manager's `skills: project-management` is the only one that survives by accident (directory-level).
- **B2 weak:** cs-product-manager orchestrates all 8 legacy skills (incl. landing-page + saas-scaffolder — scope sprawl into build work); cs-product-strategist and cs-product-manager bodies are interchangeable catalog tables, not differentiated behavior. cs-product-analyst is the cleanest (2 skills, focused).
- Verify: every path in each agent file resolves from `agents/<domain>/`; each description gains a trigger clause; cs-product-manager skill list trimmed to PM-decision skills.
## Commands
11 in scope: /rice, /okr, /persona, /user-story, /competitive-matrix, /prd, /sprint-plan, /code-to-prd (product); /sprint-health, /project-health, /retro (PM).
- **C1 pass** all 11 (frontmatter description present, usage string embedded).
- **C3 strong:** /sprint-health, /project-health, /retro, /code-to-prd — wire exact scripts with input schemas and JSON examples. Keep.
- **C3 weak:** /prd and /sprint-plan are thin prompt wrappers (output-structure bullets a bare prompt produces) — merge candidates into /rice and /user-story or beef up with script gates.
- **Stale skill-reference paths (all that cite SKILL.md):** `project-management/scrum-master/SKILL.md`, `product-team/product-manager-toolkit/SKILL.md` etc. all miss the `skills/` segment.
- **Mismatch:** /retro and /sprint-health input JSON schemas (flat per-sprint objects) don't match the scripts' actual schema (`team_info` + `sprints[]` + `retrospectives[]` per scrum-master SKILL.md) — a model following the command's example will feed the script malformed input. Verify: run each command's documented example JSON through its script; exit 0 required.
## Plugin manifests + MCP wiring
- **E1 pass:** all 6 plugin.json files schema-valid (`"skills": ["./skills"]` / `["./skills"]` canonical forms); repo-wide `check_plugin_json.py --all` green.
- **E2 fail — count/description drift:** product-skills plugin.json says "13 production-ready product skills" then enumerates 16 (incl. code-to-prd, research-summarizer, apple-hig-expert, spec-to-repo — three of which are *separate* plugins, so the description oversells what `./skills` ships). pm-skills plugin description lists 6 skills; marketplace.json says 9; disk has 9 (meeting-analyzer + team-communications + router undocumented). product-team/CLAUDE.md header says 13, lists 16, footer says "13/13". project-management/CLAUDE.md says 9, documents 6.
- **E3 pass:** all at 2.9.0, coherent with marketplace.json.
- **.mcp.json:** correct and minimal (`atlassian` → SSE `https://mcp.atlassian.com/v1/sse`). This is the only accurate piece of MCP wiring in the domain.
- **MCP documentation contradiction (cross-cutting):** four different fabricated tool-naming conventions across project-management/CLAUDE.md (`mcp__atlassian__create_issue`, `create_sprint`, `link_issue`), jira-expert (CLI flags), confluence-expert (snake_case JS), atlassian-templates (JSON blocks) — zero match the live Remote MCP surface (`createJiraIssue`, `editJiraIssue`, `searchJiraIssuesUsingJql`, `getJiraIssue`, `transitionJiraIssue`, `createConfluencePage`, `updateConfluencePage`, `getConfluencePage`, `searchConfluenceUsingCql`, `getConfluenceSpaces`, …). One canonical tool-name appendix should be written once and referenced from all four places. Verify: `grep -rE "mcp jira |mcp__atlassian__[a-z_]+|confluence_create_page|create_space\(" project-management/` returns 0 hits after fix.
audit/newgen-2026-06/productivity-markdown-html.md
+131
View File
@@ -0,0 +1,131 @@
# Domain audit: productivity/ + markdown-html/ — new-gen model optimization
Audited: 2026-06-10 · Skills: 11 · Agents: 7 · Commands: 14 · Plugins: 6 · Hooks: 2
## Scorecard
| Skill | Verdict | Top issue |
|---|---|---|
| markdown-html-orchestrator | OPTIMIZE | Stale v2.10.0 "foundation status" text instructs the model NOT to use the (now-shipped) converters |
| design-system | KEEP | Validator script exits 0 even on FAIL verdict (refusal lives in onboard.py — documented, but easy to misread) |
| md-document | KEEP | <100-line "hard rule" is prose/command-enforced, not script-enforced (parser accepted a 70-line file) |
| md-review | KEEP | — |
| md-slides | KEEP | Boundary-less 1-slide file exits 6 (no-boundary), not 5 (poster) — doc nuance |
| capture | KEEP | dump_classifier CLI shape (positional arg) not documented in SKILL.md tooling table |
| inbox-setup | KEEP | — (its agent has a phantom skills path; see Agents) |
| inbox-triage | KEEP | — (same agent issue) |
| reflect | OPTIMIZE | 3 scripts are unwired — no CLI invocations in SKILL.md, no step consumes their output |
| handoff | KEEP | "17 regex patterns" claim is actually 16; tools table lacks exact CLI invocations |
| andreessen | KEEP | — |
Verdicts: 9 KEEP · 2 OPTIMIZE · 0 REWRITE · 0 CUT-OR-MERGE
## Empirical verification results
All runs used `MARKDOWN_HTML_NO_CONFIG=1` or an isolated `HOME` to avoid touching real config. Scratch: /tmp/audit-pmh.
| # | Check | Expected | Actual | Result |
|---|---|---|---|---|
| 1 | md-document pipeline (parser → renderer → injector) on 515-line repo CLAUDE.md | valid single-file HTML | 88 KB HTML, parses clean, 16 sections, all 4 JS features injected (+5,081 B) | PASS |
| 2 | Output single-file discipline | only Google Fonts + Prism externals | external hosts: `fonts.googleapis.com`, `cdn.jsdelivr.net` (+ content links) | PASS |
| 3 | interactivity_injector idempotency | re-inject is no-op | "no-op: marker … already present", exit 0 | PASS |
| 4 | md-document <100-line refusal at script level | refuse per SKILL.md hard rule #1 | parser/renderer accepted a 70-line file (gate lives in orchestrator `route_explainer.py` + command prose `wc -l`) | PARTIAL — claim overstates script behavior |
| 5 | slide_splitter 1-slide deck → exit 5 | exit 5 | exit 5 (with `--boundary h1` single-H1, and 3-HR degenerate deck) | PASS |
| 6 | slide_splitter no-boundary → exit 6 | exit 6 | exit 6 on 120-line prose file (also on plain 1-slide file — exit 6 fires before exit 5 when no boundaries detected) | PASS |
| 7 | md-slides happy path (5 slides, 3 with notes) | working deck | 10.8 KB single-file HTML, 5 `<section class="slide">`, keydown handlers, 1 `@media print`, 60% notes coverage reported | PASS |
| 8 | md-review diff_parser on real ```diff block | hunks JSON | 1 file / 1 hunk parsed; extractor found 2 annotations (1 BLOCKER, 1 NIT) | PASS |
| 9 | review_html_renderer without `--reviewer` → exit 3 | exit 3 | exit 3, "A code review must name a human reviewer" | PASS |
| 10 | review_html_renderer with 0 hunks → exit 4 | exit 4 | exit 4, "route to md-document instead" | PASS |
| 11 | LGTM approval capture | approvals counted | standalone `LGTM` line → 1 approval (regex requires standalone line; "LGTM otherwise." is not counted — by design) | PASS |
| 12 | brand_palette_validator refuses AA-failing palette | FAIL verdict | `--text #CCCCCC --bg #FFFFFF` → FAIL 1.61:1; but script exit 0 (verdict-only); save refusal is onboard.py | PASS (with caveat) |
| 13 | onboard.py exit 4 on AA-fail save | exit 4 | exit 4, "refusing to save: WCAG AA contrast failed" | PASS |
| 14 | onboard.py exit 3 on empty/unwritable output dir | exit 3 | exit 3 on empty path (unwritable case untestable as root — `os.access` always true; code path present at line 226) | PASS |
| 15 | orchestrator <100-line refusal | REFUSE | exit 3, Shihipar citation, design-system status surfaced | PASS |
| 16 | orchestrator silent-route on review doc | ROUTE_SILENTLY → md-review | score 13 vs runner-up 1, routed silently | PASS |
| 17 | orchestrator ambiguity → ASK_USER | one question | CLAUDE.md scored slides 18 / document 15 → ASK_USER with recommended answer (correct: CLAUDE.md is full of `---` HRs) | PASS |
| 18 | handoff redaction_linter on fake AWS key (strict) | exit 1, block | exit 1, `[high] aws_access_key`, fix suggestion + whitelist tip | PASS |
| 19 | redaction_linter whitelist `<!-- handoff:allow secret -->` | exit 0 | exit 0, "OK: no findings" | PASS |
| 20 | "17 patterns" claim | 17 Pattern() defs | **16** Pattern() defs (aws×2, github, openai, anthropic, slack, google, stripe, private-key, jwt, env-assign, db-conn, bearer, email, phone, url-token) | FAIL (off-by-one in docs) |
| 21 | handoff hooks | stdin-safe, env-disable | SessionStart: exit 0 disabled + exit 0 no-handoff; SessionEnd prints reminder, exit 0; hooks.json wires both via `${CLAUDE_PLUGIN_ROOT}` | PASS |
| 22 | andreessen market_first_evaluator --sample | verdict + weights | BUILD-POUR-FUEL, market weighted 0.55 (contribution 4.4) | PASS |
| 23 | andreessen kill gate: market 3.0, team/product 10 | KILL despite 6.15 composite | KILL-OR-REPICK-MARKET + explicit "trap" note that team/product cannot override sub-4 market | PASS |
| 24 | anti_todo_card 6th must-do | reject | exit 2, "the cap IS the discipline" | PASS |
| 25 | operating prompt operationalized | posture table | references/operating_prompt.md: verbatim prompt + 6-row instruction→behavior mapping + binding confidence-level discipline + "what this is NOT" | PASS |
| 26 | inbox-triage draft_safety_validator | exit 1 on send-shaped call | `--sample-fail` exit 1 / `--sample-pass` exit 0 | PASS |
| 27 | all 22 productivity scripts `--help` | exit 0 | 22/22 exit 0 | PASS |
Tally: **24 PASS · 1 PARTIAL · 1 FAIL** (plus 1 caveat). The domains' empirical claims are overwhelmingly real.
## Domain-level findings
1. **Staleness cascade in markdown-html/ (the one real defect).** The domain shipped complete at v2.10.3, but three files still describe the v2.10.0 foundation: `markdown-html/CLAUDE.md` (skills table marks md-document/review/slides "v2.10.1"), `markdown-html/README.md` ("Status — v2.10.0 (foundation)", converters "v2.10.1 (next PR)"), and the orchestrator SKILL.md ("Until v2.10.1, the orchestrator's job stops at step 4 — … lets Claude do the rendering inline"). A new-gen model following the orchestrator SKILL.md today is **instructed to bypass the shipped converters** and hand-render. This is an A6 failure with behavioral consequence, not cosmetic.
2. **Refusal gates split between scripts and prose — mostly fine, but SKILL.md wording overclaims twice.** md-document's "Hard rule 1: Refuses input < 100 lines" and design-system's validator both read as script-level gates; in reality the 100-line gate lives in the orchestrator's `route_explainer.py` and in the converter commands' `wc -l` instruction, and the palette refusal lives in `onboard.py` (validator exits 0 on FAIL verdict). Direct script invocation skips both. Acceptable for a model that follows the commands, but the prose should say where each gate is enforced.
3. **A3 weakness across productivity/: tool tables without exact CLI invocations.** reflect, handoff, and capture (dump_classifier) list scripts in a table but never show how to call them — I had to discover arg shapes by trial (`redaction_linter.py FILE` positional, not `--file`; `bias_pattern_detector.py --conversation`). markdown-html/ does this right (every SKILL.md has copy-pasteable invocations); productivity/ should match.
4. **Counter/spec drift.** Redaction patterns 16 vs claimed 17 (root CLAUDE.md v2.8.2 notes). reflect's spec is `megaprompts/02-reflect-megaprompt.md` per SKILL.md + plugin.json, but root CLAUDE.md v2.7.0 notes call reflect "megaprompt 08". Minor, but counters are this repo's brand — keep them true.
5. **Trigger quality is uniformly strong (A1).** Every skill in scope has concrete trigger phrases, third-person descriptions, refusal conditions in the description itself, and "distinct from" disambiguation. This is the best trigger discipline of any domain pattern observed; no action needed.
6. **Context economy is good but markdown-html SKILL.md bodies duplicate the domain CLAUDE.md hard rules ~3×** (domain CLAUDE.md, SKILL.md "Hard rules", command "Pre-flight gates"). Tolerable since skills ship standalone, but the duplication is what made the staleness cascade possible — single-source the version/status table.
## Per-skill findings
### markdown-html-orchestrator — OPTIMIZE
- **Verdict:** OPTIMIZE (targeted edits; routing logic and scripts are excellent and fully verified)
- **Issues:**
1. SKILL.md "Foundation status (v2.10.0)" paragraph + Step 5 ("Until v2.10.1 … lets Claude do the rendering inline") + Output-artifacts table ("v2.10.1" status column) instruct the model to bypass shipped converters. Delete the transitional text.
2. Frontmatter `version: 2.10.0` while plugin is 2.10.3.
3. `markdown-html/CLAUDE.md` skills table and `README.md` status section carry the same stale v2.10.0 framing (fix together).
4. Pipeline snippets use `skills/markdown-html-orchestrator/...` relative paths while Step-1 uses repo-rooted `markdown-html/skills/...` — pick one convention.
- **Verify:**
- `grep -c "v2.10.1" markdown-html/skills/markdown-html-orchestrator/SKILL.md` returns 0
- `grep -c "foundation" markdown-html/README.md markdown-html/CLAUDE.md` returns 0 stale-status hits (status table lists all 5 skills "✓ live")
- `printf '# s\n' > /tmp/s.md && python3 markdown-html/skills/markdown-html-orchestrator/scripts/doctype_classifier.py --input /tmp/s.md --output json | python3 .../route_explainer.py; test $? -eq 3` (refusal stays green)
- review-shaped ≥100-line input still yields `ROUTE_SILENTLY -> md-review`
### reflect — OPTIMIZE
- **Verdict:** OPTIMIZE (the prompt body is strong; the script layer is dead weight as wired)
- **Issues:**
1. A3: no CLI invocations anywhere in SKILL.md for the 3 scripts; no workflow step consumes their output. For an in-conversation reflection skill the model cannot trivially produce a transcript file, so `bias_pattern_detector.py --conversation FILE` and `conversation_depth_analyzer.py` have no realistic input path described.
2. A4: `directional_recommendation_validator.py` is the natural verification loop (assert output ends Continue/Pivot/Pause) but is never invoked in the workflow. Wire it: "pipe your draft reflection through the validator before sending; exit 0 required."
3. Either wire all 3 scripts with exact invocations + an input-capture step, or cut detector/analyzer and keep only the validator (A7).
- **Verify:**
- SKILL.md "Tooling" section contains ≥1 fenced `python3 …` invocation per retained script
- `printf 'analysis...\nContinue. Because X and Y are verified.\n' | python3 productivity/reflect/skills/reflect/scripts/directional_recommendation_validator.py -` (or documented file form) exits 0; output missing a closing recommendation exits non-zero
- `python3 .../bias_pattern_detector.py --sample --output json` exits 0 with keys `biases_detected`, `biases_clear`, `details` (if retained)
## KEEP-verdict verification criteria
- **design-system:** `HOME=$(mktemp -d) python3 markdown-html/skills/design-system/scripts/onboard.py --defaults` exits 0; `--set brand.text='#CCCCCC' --set brand.bg='#FFFFFF'` exits 4; `--set default_output_dir=` exits 3; `config_loader.py --status` reports `setup_completed: true` after defaults.
- **md-document:** full 3-script pipeline on a ≥100-line file exits 0×3 and produces HTML whose only external hosts are `fonts.googleapis.com` + `cdn.jsdelivr.net`; second injector run prints `no-op` and exits 0. (Optimization PRs must also fix Hard-rule-1 wording to name where the 100-line gate is enforced.)
- **md-review:** renderer without `--reviewer` exits 3; with 0-hunk input exits 4; with valid input exits 0 and output contains the reviewer name + per-severity counts; standalone `LGTM` line yields `approvals: 1`.
- **md-slides:** `slide_splitter.py` exits 5 on a boundary-detected 1-slide deck, 6 on boundary-less input, 0 on a 5-slide HR deck; rendered deck contains 5 `class="slide"` sections, `addEventListener`, and `@media print`.
- **capture:** `dump_classifier.py <file>` and `--sample` exit 0; `workspace_inventory.py --root . --keywords "a,b"` exits 0 (documented invocation stays true); `complexity_estimator.py --sample small` recommends compressed format.
- **inbox-setup:** `kb_validator.py --sample` exits 0 with verdict PASS and 7-file contract checks; SKILL.md section count stays 8 with S4 skip-logic intact.
- **inbox-triage:** `draft_safety_validator.py --sample-fail` exits 1, `--sample-pass` exits 0; SKILL.md states DRAFTS-ONLY in ≥3 places; `search_window_calculator.py --help` exits 0.
- **handoff:** `redaction_linter.py <file-with-AKIA-key>` exits 1 strict / 0 with `<!-- handoff:allow secret -->`; `echo '{}' | HANDOFF_SESSIONSTART=0 python3 hooks/session_start.py` exits 0; `handoff_self_check.py --sample` exits 0; fix the "17 patterns" claim to 16 (or add the 17th) wherever it appears.
- **andreessen:** `market_first_evaluator.py --size 3 --growth 3 --timing 3 --pull 3 --team 10 --product 10` verdict is KILL-OR-REPICK-MARKET; `anti_todo_card.py --new --must-do a b c d e f` exits 2; `references/operating_prompt.md` retains the verbatim prompt + 6-row posture table.
## Agents
7 agents. One real bug, otherwise differentiated and non-boilerplate (B2/B3 pass — capture's 210-line persona, inbox pair's halting rules, and andreessen's binding voice could not be swapped unnoticed).
- **BUG — `cs-inbox-setup.md` and `cs-inbox-triage.md` declare `skills: engineering/email/skills/inbox-setup` / `...inbox-triage`. The skills live at `productivity/email/skills/...`; `engineering/email/` does not exist.** Phantom path (B1). Fix: `skills: productivity/email/skills/inbox-setup` (resp. `-triage`).
- B1 trigger phrasing: only `cs-handoff-author` uses explicit "Invoke when…" language. The other six describe the persona but not when to fire; cheap win to prepend "Use when…" to each description.
- `cs-markdown-html-orchestrator` (model: sonnet) carries the same stale assumption indirectly via the skill it wraps — no edit needed once the SKILL.md is fixed.
- Frontmatter style is inconsistent across the set (`tools: [..]` vs `tools: "..."` vs quoted lists; `model: opus`/`sonnet`/`inherit`) — harmless but worth normalizing in an optimization PR.
## Commands
14 commands. All have accurate descriptions (C1). All orchestrate tools or enforce gates a bare prompt would not (C3): the six markdown-html commands embed pre-flight refusal gates + exact pipelines; `/cs:handoff` enforces checklist→linter→save ordering; `/cs:inbox-*` enforce the one-question-per-turn and DRAFTS-ONLY disciplines; `/cs:andreessen` + `/cs:pmf-check` bind verdict tools.
- C2: markdown-html commands + handoff pair carry `argument-hint`; the 6 productivity megaprompt-era commands (andreessen, pmf-check, capture, inbox-setup, inbox-triage, reflect) embed usage in the body instead of `argument-hint` frontmatter — minor normalization.
- `/cs:grill-markdown-html` and `/cs:design-system` are genuinely distinct surfaces (grill vs route vs onboard); no merge candidates.
## Plugin manifests
6 plugins, all schema-valid (repo-wide `check_plugin_json.py --all` green), all using canonical `./`-prefixed skills arrays.
- **markdown-html-skills (2.10.3):** description counts verified — 15 tools (3×5 ✓), 15 references (3×5 ✓), 4 assets (1 schema + 3 templates ✓), 5 skill paths ✓. E2/E3 pass. Best manifest in scope.
- **Productivity five (all 2.9.0):** coherent with their marketplace.json entries (E3 pass), though frozen at 2.9.0 while the repo is at 2.10.3 — per repo policy versions should track releases; bump at next touch.
- **Name drift (low):** marketplace entry names differ from plugin.json `name` for four plugins — `capture-skill`/`capture`, `email-pair`/`email`, `reflect-skill`/`reflect`, `handoff-productivity`/`handoff`. If intentional (ClawHub slug conflicts), document it; otherwise align.
- `handoff` plugin correctly omits a `hooks` key in plugin.json while shipping `hooks/hooks.json` — Claude Code's auto-discovery convention; hooks verified working via `${CLAUDE_PLUGIN_ROOT}` commands.
## Hooks
2 hooks (handoff SessionStart + SessionEnd). Both stdlib-only, fail-open (`sys.exit(0)` on import error — "hook must never break a session"), env-disable verified (`HANDOFF_SESSIONSTART=0` / `HANDOFF_SESSIONEND=0`), 12,000-char body cap on injected handoff prevents context blowout. D4 by-design stdin exception documented in-file. PASS.
audit/newgen-2026-06/research.md
+133
View File
@@ -0,0 +1,133 @@
# Domain audit: research/ + research-ops/ — new-gen model optimization
Audited: 2026-06-10 · Skills: 13 (8 research/ + 5 research-ops/) · Agents: 9 · Commands: 14 · Plugins: 9
## Scorecard
| Skill | Verdict | Top issue |
|---|---|---|
| research-ops/research-ops-skills (orchestrator) | KEEP | Routing is prose-only (no classifier script like research/ ships) — acceptable, but inconsistent with sibling |
| research-ops/clinical-research | KEEP | — (model citizen; all hard rules verified in tool output) |
| research-ops/research-finance | KEEP | — (capex router never auto-decides; named owner verified) |
| research-ops/market-research | KEEP | — (both-methods TAM + triangulation flag verified) |
| research-ops/product-research | KEEP | — (INSIGHT vs ANECDOTE gate verified) |
| research/research (orchestrator) | OPTIMIZE | Single-keyword signals ("funding", "fda", "patent", "grant") cause weak-match misroutes; 319-line body restates generic search methodology |
| research/pulse | OPTIMIZE | Unauthenticated reddit.com/search.json is post-2023-API fragile; duplicated Agent Integrity boilerplate |
| research/litreview | OPTIMIZE | Hard Consensus-MCP dependency with no free-API fallback; phantom `scripts/office/validate.py` |
| research/grants | OPTIMIZE | Phantom `scripts/office/validate.py`; otherwise the strongest skill in the pack (RePORTER POST, dynamic FY, correct NIH receipt dates) |
| research/dossier | OPTIMIZE | Phantom `scripts/office/validate.py`; 318-line body |
| research/patent | OPTIMIZE | Phantom validate.py; `patents.uspto.gov/patent/...` hyperlink pattern is wrong (PPS lives at ppubs.uspto.gov) |
| research/syllabus | OPTIMIZE | Phantom validate.py; bundled JS requires npm `docx` (fails clean, but install path undocumented in plugin) |
| research/notebooklm | REWRITE | Hardcoded UI inventory of a fast-moving Google product has already rotted; description (8 Studio types) contradicts body (9) |
Verdict counts: **KEEP 5 · OPTIMIZE 7 · REWRITE 1 · CUT-OR-MERGE 0**
## Domain-level findings
1. **Phantom tool path across 5 research/ skills (A3 fail).** `python scripts/office/validate.py <docx>` is referenced as the DOCX validation step in litreview, grants, dossier, patent, and syllabus SKILL.md (plus litreview's README/agent/command and two reference docs). The file exists nowhere in the repo (`find -path "*office/validate.py"` → empty). Every DOCX workflow ends with an instruction the model cannot execute. One fix (ship the validator or delete the step) clears 5 skills at once.
2. **research-ops/ hard rules are genuinely operationalized — all sample runs passed.** `sample_size_estimator.py --sample` prints the "ESTIMATE ONLY — confirm with a biostatistician" banner + assumptions block + named-owner requirement; `market_sizer.py --sample` prints top-down AND bottoms-up TAM/SAM/SOM, a 73.4% divergence figure, and a "TRIANGULATION FAILED" flag; `insight_synthesizer.py --sample` promotes the 3-participant cluster to INSIGHT and flags 1- and 2-participant clusters as ANECDOTE; `capex_vs_opex_router.py --sample` routes all three verdicts to "R&D Finance Controller (+ External Auditor)"; `phase_gate_scorer.py --sample --output json` emits `verdict: GO` with a 3-name owner chain. Config consumption is real (tools import `config_loader`, CLI overrides, `RESEARCH_OPS_NO_CONFIG=1` bypass). This domain is the template the research/ pack should be refactored toward.
3. **research/ context economy is poor and boilerplate is forked, not shared.** SKILL.md bodies run 251–319 lines each; the "Agent Integrity Rules (Research-Pack Convention)" block is repeated near-verbatim in 7 files; `citation_tracker.py` exists as **6 divergent copies** (217–303 lines, all different md5s, 1,556 lines total). Per-skill self-containment is repo policy, but the variants have already drifted — a bug fix in one will not propagate. A shared reference + per-skill thin wrapper would cut ~40% of the pack's context weight.
4. **The repo's own checklist flags are partly a phrasing artifact, partly real.** Re-ran `skill_review_checklist_runner.py` on all 13: research-ops = 4/6 across the board (only fails the under-100-lines rule + the light 'skill'/'tool' terminology heuristic); research/ = 2–4/6. The "Missing trigger" failures on 7 of 8 research/ skills are because descriptions say `Triggers: 'pulse on [topic]'…` instead of the validator's `Use when…` pattern — the descriptions themselves are substantively rich (A1 passes on content, fails the repo gate on phrasing). The notebooklm 2/6 and research 2/6 KNOWN scores reproduce; research additionally trips the time-sensitive check ("in 2026").
5. **External-surface fragility is concentrated in research/.** Consensus MCP is a hard dependency for litreview, syllabus, and grants Phase 2A with zero fallback to free academic APIs (PubMed E-utilities, OpenAlex, Semantic Scholar — all keyless); pulse leans on unauthenticated Reddit JSON endpoints that Reddit increasingly blocks from datacenter IPs; notebooklm hardcodes a Google SPA's UI inventory. New-gen models with native WebSearch/WebFetch make free-API fallbacks cheap to specify — the skills predate that assumption.
6. **The orchestrator routing claim verifies.** `classifier.py` SIGNALS dict matches the SKILL.md table phrase-for-phrase; live tests: 3-signal litreview question → `route_to: litreview`, "research Microsoft" → `fallback` (as the SKILL.md explicitly promises), "dossier on Acme Corp for due diligence" → `dossier` (2 signals). The ≥2-signal threshold, single-weak-match rule, and fallback rule are all implemented. The followability problem is precision, not existence (see per-skill).
7. **All 48 Python scripts across both domains pass `--help` exit 0**; stdlib-only confirmed. The one non-Python script (`generate_reading_list.js`) fails without `npm install docx` but fails with a clear actionable message (acceptable, documented).
8. **Stray `__pycache__/*.pyc` committed under 4 research-ops script dirs** — repo hygiene, should be gitignored.
## Per-skill findings
### research/research (orchestrator) — OPTIMIZE
- Single-keyword signals over-trigger: "funding", "fda", "grant" (grants), "patent", "invention" (patent), "curriculum" (syllabus) each score 1 alone → the "single weak match" rule silently routes e.g. "research FDA approval trends" to grants. Multi-word phrases route reliably; bare nouns don't.
- 319 lines; Phase 3b fallback (decompose → search → synthesize → cite) restates what a frontier model does unprompted — keep the budget numbers + audit-log contract, cut the how-to prose.
- "Waits 1 turn… or auto-proceeds after 5s" is an un-executable affordance (the model cannot wait wall-clock time); checklist also flags "in 2026" as time-sensitive.
- Description lacks `Use when` phrasing → fails repo A1 gate (2/6 known score reproduces).
- Verify: `python3 scripts/classifier.py --question "research Microsoft" --output json` → `route_to: "fallback"`.
- Verify: `python3 scripts/classifier.py --question "literature review on PICO meta-analysis" --output json` → `route_to: "litreview"`, ≥2 signals.
- Verify: bare-noun precision test added: "research FDA approval trends" must NOT route to grants (after signal-list fix).
- Verify: checklist runner item 1 (trigger) and item 3 (time-sensitive) pass.
### research/pulse — OPTIMIZE
- Reddit phase depends on unauthenticated `reddit.com/search.json`; since the 2023 API changes these endpoints are routinely 403'd from non-browser/datacenter clients. Fallbacks exist (`raw_json=1`, subreddit-restricted) but a "Reddit fully blocked → degrade to Web-phase reddit site: search" path is missing.
- Hardcoded trusted-publisher `site:` list (NYT/WSJ/Wired/Verge/TechCrunch) is US-tech-centric and will silently skew non-tech topics.
- 258 lines incl. duplicated Agent Integrity block; checklist fails under-100-lines + terminology.
- Verify: `python3 scripts/time_window_calculator.py --window 30d` exits 0 and emits both HN `created_at_i` timestamp and Reddit `t=month`.
- Verify: `python3 scripts/citation_tracker.py --help` exits 0; session file lands at `~/.pulse_sessions/`.
- Verify: SKILL.md documents an explicit all-Reddit-blocked degradation path.
### research/litreview — OPTIMIZE
- Hard Consensus-MCP dependency; no fallback to free academic APIs (PubMed E-utilities / OpenAlex / Semantic Scholar are keyless) — the skill is inert in any harness without that one MCP.
- References phantom `python scripts/office/validate.py output.docx` (file does not exist anywhere in repo).
- Plan-tier detection parses marketing copy ("Showing top 10" / "upgrade") — brittle heuristic; will mis-detect when Consensus rewords.
- Verify: `python3 scripts/framework_recommender.py --help` and `cross_search_aggregator.py --help` exit 0.
- Verify: no reference to `scripts/office/validate.py` remains (grep returns empty) OR the validator ships.
- Verify: SKILL.md names at least one free-API fallback for the no-Consensus case.
### research/grants — OPTIMIZE
- Phantom `scripts/office/validate.py` in Phase 4 (only blocking edit — domain expertise is otherwise the best in the pack: RePORTER v2 POST templates, NOSI URL pattern, scope-aware mechanism matrix, dynamic FY window, NIH standard receipt dates all check out).
- Phase 2A (5 Consensus searches) has no free fallback; RePORTER core works without it but the SKILL treats Consensus as mandatory.
- Description fails the `Use when` gate (phrasing only).
- Verify: `python3 scripts/fiscal_year_calculator.py --output json` → `current_fy` correct for today's date (Oct 1 boundary), 4-year window.
- Verify: `python3 scripts/mechanism_matcher.py --career-stage early_career --prelim-data pilot --environment r01_eligible --scope single_site --output json` → shortlist excludes R01, includes R21/K-series.
- Verify: phantom validate.py reference removed or implemented.
### research/dossier — OPTIMIZE
- Phantom `scripts/office/validate.py` in Phase 10; 318-line body (longest in pack) with duplicated integrity boilerplate.
- The ≥30% disconfirming-evidence rule, source-tier tagging, and mandatory-hypothesis gate are excellent new-gen design (forces the model out of confirmation mode) — preserve verbatim.
- Glassdoor/Comparably scraping listed as a source will be blocked in practice; "degrade gracefully" is stated but the degraded output shape isn't.
- Verify: `python3 scripts/disconfirming_evidence_balance.py --help` exits 0; given a session with <30% disconfirming queries it returns a non-zero/warn signal.
- Verify: `python3 scripts/source_tier_classifier.py --help` exits 0; sec.gov → primary, a substack URL → tertiary.
- Verify: phantom validate.py reference removed or implemented.
### research/patent — OPTIMIZE
- Phantom `scripts/office/validate.py` in Phase 7.
- DOCX styling section gives `https://patents.uspto.gov/patent/...` as the USPTO hyperlink pattern — that host pattern is wrong (the skill's own source list correctly says `ppubs.uspto.gov`); links generated from it will 404.
- Sub-use-case routing, date discipline (filing/priority/publication/grant per use case), and CPC class follow-up are real practitioner expertise — keep.
- Verify: `python3 scripts/sub_use_case_router.py --sub-use-case novelty --jurisdictions "" --risk strict --known-art "US10000000B2"` exits 0 and emits a 5-8 query plan.
- Verify: `python3 scripts/family_resolver.py --help` exits 0.
- Verify: no `patents.uspto.gov/patent/` literal remains in SKILL.md.
### research/syllabus — OPTIMIZE
- Phantom `python scripts/office/validate.py` in Phase 6; Consensus-MCP hard dependency (same fix as litreview).
- `generate_reading_list.js` requires npm `docx`; fails with a clear message (verified) but neither SKILL.md Portability note nor plugin docs give the install one-liner next to the invocation.
- Applied-domain weaving + Bloom higher-order validator are genuinely non-obvious value — keep.
- Verify: `node scripts/generate_reading_list.js --help` without docx installed exits non-zero with the "npm install docx" message (graceful-fail contract).
- Verify: `python3 scripts/discussion_question_validator.py --help` and `topic_grouper.py --help` exit 0.
- Verify: phantom validate.py reference removed or implemented.
### research/notebooklm — REWRITE
- Freshness rot on a fast-iterating Google SPA: the hardcoded Studio inventory (9 types incl. "Table of Contents") no longer matches the product — NotebookLM added Video Overviews (2025) and Flashcards/Quiz, and reorganized report-style outputs; none appear anywhere in skill, scripts, or references (moderate-high confidence; needs re-verification against the live UI).
- Internal inconsistency: frontmatter description lists 8 Studio types, body Action 3 lists 9 — the plugin.json mirrors the stale 8.
- Known 2/6 checklist score reproduces: no `Use when` trigger phrasing, 290 lines, zero code blocks (no concrete invocation examples).
- Structure is salvageable (Step-0 environment gate, fire-and-notify async table, screenshot-first, find()-before-click are all sound discipline; `async_action_classifier.py` works); the rot-prone content is the hardcoded UI inventory + timing estimates. Rewrite to discover output types from the live Studio panel screenshot instead of enumerating them, add a "verified against NotebookLM as of <date>" maintenance marker, fix the 8-vs-9 contradiction.
- Verify: `python3 scripts/async_action_classifier.py --action "audio overview"` → `FIRE_AND_NOTIFY`; `--action "add source"` → wait verdict.
- Verify: frontmatter description and Action 3 list enumerate the same set (or neither enumerates).
- Verify: SKILL.md contains ≥1 fenced concrete-example block and a `Use when`-style trigger; checklist item 5 passes.
- Verify: a dated "UI verified" marker exists and is < 6 months old at release time.
## KEEP-verdict verification criteria
- **research-ops-skills (orchestrator):** SKILL.md signal table lists exactly 4 lanes matching the 4 sub-skill folder names; `grep -c "Never silently chain" SKILL.md` ≥ 1; every `skills/<sub-skill>/scripts/onboard.py` path it references resolves from `research-ops/`.
- **clinical-research:** `python3 scripts/sample_size_estimator.py --sample` exits 0, output contains "ESTIMATE" banner and `n_group1_with_dropout` > `n_group1_raw`; `phase_gate_scorer.py --sample --output json` → `verdict` ∈ {GO, GO-WITH-CONDITIONS, REDESIGN, NO-GO} and `named_owners` non-empty; `endpoint_selector.py --sample` flags the unvalidated surrogate below PRIMARY.
- **research-finance:** `capex_vs_opex_router.py --sample --standard ifrs` exits 0, every item carries `route to:` a named owner and the "DECISION SUPPORT ONLY" banner prints; `burn_runway_tracker.py --sample --output json` → `runway_months_approx` float + per-milestone `verdict`.
- **market-research:** `market_sizer.py --sample` prints BOTH `[top-down]` and `[bottoms-up]` TAM/SAM/SOM lines, a divergence %, and "TRIANGULATION FAILED" when delta > tolerance; `sample_size_planner.py --population 62000 --confidence 0.95 --moe 0.05` exits 0 with FPC-corrected n; `segmentation_scorer.py --sample` drops the solopreneur slice.
- **product-research:** `insight_synthesizer.py --sample --min-sources 3` promotes the 3-source cluster to INSIGHT and labels 1–2-source clusters ANECDOTE; `saturation_planner.py --method thematic --segments 3` emits a confidence label and the "not a power calculation" disclaimer; `study_designer.py --goal evaluative --stage live` redirects live A/B to experiment-designer.
## Agents
9 agents total (8 in `research/<plugin>/agents/`, 1 in `research-ops/agents/`). **All pass B2/B3 strongly** — each persona has distinct refusal behaviors that change outcomes (cs-dossier refuses without a hypothesis; cs-patent refuses without a sub-use-case; cs-research surfaces routing + override; cs-research-ops-orchestrator demands method-before-number) and verbatim voice lines, not adjective swaps. B1 issues:
- research/ agent descriptions describe behavior but lack `Use when`/`Use PROACTIVELY` trigger phrasing (same artifact as the SKILL.md gate).
- All 8 research/ agents pin `model: opus` — for a deterministic-routing front door (cs-research) sonnet would do; cost note only.
- research/ agents carry non-standard `skills:`/`domain:` frontmatter keys (harmless, but not part of the agent schema the repo's other domains use).
- cs-research-ops-orchestrator is the cleanest: standard frontmatter, `model: sonnet`, Skill tool wired for fork-routing.
## Commands
14 total: 8 `/cs:*` in research/ plugins, 6 in research-ops/. All pass C1 + C3 (each orchestrates intake gates, tool sequences, and refusal rules a bare prompt would not enforce). Findings:
- **research-ops commands are the better pattern:** proper `argument-hint:` field + explicit `$ARGUMENTS` interpolation + per-command three-tool workflow. `/cs:grill-research-ops` adds real value (docs-anchored one-question-at-a-time gate before any sub-skill).
- **research/ commands** embed the argument shape inside the description string and omit `argument-hint` / `$ARGUMENTS` (C2 partial) — they read as documentation pages for the agent rather than parameterized commands. Targeted fix: add `argument-hint` + `$ARGUMENTS` block to all 8.
- Mild redundancy: `/cs:research` + 6 specialist commands + the router skill itself is three entry points to the same routing logic; acceptable, but the specialist commands should state "or just use /cs:research" to avoid user confusion.
## Plugin manifests
All 9 pass `scripts/check_plugin_json.py --all` (E1). Versions uniformly 2.9.0 and present in marketplace.json (E3). Issues:
- **Name mismatch (E3 minor):** marketplace entry `research-orchestrator` points at `./research/research` whose plugin.json `name` is `research`. Intentional slug-disambiguation per ClawHub rules, but the inconsistency is undocumented in either file.
- **Stale content drift (E2):** `notebooklm` plugin.json description carries the 8-type Studio list (mirrors the stale SKILL.md frontmatter) — fix together with the notebooklm rewrite.
- research-ops single-domain-plugin description accurately enumerates the 4 sub-skills + orchestrator; matches contents.
- Hygiene: committed `__pycache__/*.pyc` under `research-ops/skills/{clinical-research,market-research,product-research,research-finance}/scripts/` should be removed/ignored.
audit/pm-product-agentic-2026-07/00-MASTER.md
+123
View File
@@ -0,0 +1,123 @@
# Master report — Product & PM agentic-loop audit + domain harness upgrade
**Audited:** 2026-07-03 · **Branch:** `claude/pm-audit-agentic-loops-jxurlq` ·
**Scope:** both product/project domains — `product-team/` (17 skills) and
`project-management/` (9 skills) — deep-audited on quality AND scored on the
**agentic-readiness** rubric established by the engineering audit
([../engineering-agentic-2026-07/](../engineering-agentic-2026-07/00-MASTER.md)).
Plus: both domains upgraded into **agent harnesses** — fork-orchestrators with
deterministic routers, reusable loops, machine-checkable verification gates, and
integration with the repo-wide `engineering/agent-harness` framework.
**Method:** (1) two parallel deep-dive agents read every SKILL.md, smoke-tested all 31
scripts, and cross-referenced agents/commands/manifests; (2) one explorer mapped the
repo's harness conventions (loop-library contract, agent-harness state machine,
fork-orchestrator pattern) so the upgrade reuses rather than reinvents; (3) one research
agent web-verified the 2024–2026 PM/product/harness canon
([research-digest.md](research-digest.md)).
---
## 1. The two questions
The June 2026 audit asked: does each skill earn its context window? This audit asks the
engineering follow-up question for these two domains: **can an agent pick up a goal here
and drive it to a verified close?** — and additionally: **what should these domains
teach that the 2024–2026 canon now demands?**
([improvement-fields.md](improvement-fields.md) answers the second.)
## 2. Combined scorecard (26 skills, post-PR)
| Class | product-team | project-management | Total | Meaning |
|---|---|---|---|---|
| **HARNESS-READY** (≥9, AR4≥1, AR5≥1) | 2 | 1 | **3** | An agent can loop this today |
| **LOOP-CAPABLE** (6–8) | 4 | 4 | **8** | One or two additions away |
| **TOOL-ONLY** (3–5) | 11 | 3 | **14** | Good tools, no loop spine |
| **PROSE-ONLY** (0–2) | 0 | 1 | **1** | Needs structural rebuild |
Pre-PR both domain routers were PROSE-ONLY (score ≤ 1) and neither domain had a single
`context: fork`, forcing question, iteration cap, or `/cs:*` command — they predate every
v2.8+ convention. The weakest dimensions mirror engineering exactly: **AR5 loop
discipline** (zero caps anywhere pre-PR) and **AR1 goal intake** (most skills accept any
input silently).
## 3. The three biggest findings
1. **The MCP↔analytics gap (project-management).** The domain bundles a live Jira MCP
and ships real analytics tools, with no data path between them — sprint health and
velocity ran on hand-typed JSON. Fixed: `jira_snapshot_bridge.py` converts saved
`searchJiraIssuesUsingJql` results into the scrum-master schema (verified end-to-end
into velocity_analyzer) and computes the four Kanban-Guide-2025 flow metrics + seeded
Monte Carlo forecasts the domain never had.
2. **Verification exists but nothing binds it (both domains).** spec-to-repo's
validator, code-to-prd's golden outputs, scrum-master's pinned fixtures,
atlassian-admin's 7 VERIFY steps — good gates, all optional, none looped. Fixed at
the orchestration layer: plans are gated before execution and closes are refused
(exit 4) while tasks are unverified/unwaived; per-skill binding is follow-up F3.
3. **The canon moved (both domains).** No continuous-discovery cadence, no OST
discipline, no AI-feature evals, no flow metrics, no probabilistic forecasting, no
agentic-delegation governance, and 60 of 64 reference files cite zero sources. This
PR ships the two highest-leverage tool fields per domain plus six cited reference
docs; the remaining 14 fields are enumerated with tool specs in
[improvement-fields.md](improvement-fields.md).
## 4. What this PR ships: two domain harnesses
Both prose routers were rebuilt as `context: fork` orchestrators that plug into
`engineering/agent-harness` (manifests regenerated; both orchestrators now score all
five `agentic_signals`):
**project-management → `pm-skills`** — the *delivery loop*:
`pm_goal_router.py` (8 lanes, exit 0/2/3 — route/ask/refuse) ·
`jira_snapshot_bridge.py` (MCP snapshot → flow metrics | sprint schema; SLE conformance,
aging-WIP alerts, `--forecast` Monte Carlo, refuses thin history) ·
`delivery_loop_gate.py` (delegation governance G1–G6: human owner, reviewer for agent
tasks, machine-checkable acceptance, evidence-before-done, close refusal,
exhausted-budget-is-escalation). Five reusable PM loops (sprint-flow, health,
retro-action, RAID-hygiene, comms) documented with terminal states. Agent
`cs-pm-orchestrator`; commands `/cs:pm`, `/cs:grill-pm`, `/cs:pm-loop`.
**product-team → `product-skills`** — the *discovery loop*:
`product_goal_router.py` (16 lanes incl. the 4 standalone plugins) ·
`discovery_cadence_tracker.py` (Torres weekly-habit scoring: streak, coverage, outcome
linkage, test throughput → health 0–100 with named gaps and a `next_loop_action`) ·
`ost_linter.py` (O1–O5: measurable outcome root, needs-not-features, ≥2 solutions per
target, tests per solution, no orphan solutions — exit 2 blocks the tree from driving a
roadmap). Graduation stop-states hand validated assumptions to experiment-designer/PRD.
Agent `cs-product-orchestrator`; commands `/cs:product`, `/cs:grill-product`,
`/cs:product-loop`.
Also fixed: the two CLI-noncompliant product tools (`user_story_generator.py`,
`persona_generator.py` — real argparse `--help`, seeded determinism, backward-compatible
positionals); domain CLAUDE.md counters; plugin manifests + marketplace descriptions.
All 8 new/changed tools pass `--help` and `--sample`; fixtures pinned
(`expected_flow_metrics.json`; sample OST with two planted violations). Every design
decision traces to the loop-library contract, the agent-harness invariants (locked
gates, evidence-before-status, budgets-as-terminal-states), and the cited canon.
## 5. Per-domain reports
- [product-team.md](product-team.md) — 17 skills, AR table, 7 domain findings,
executable verification criteria.
- [project-management.md](project-management.md) — 9 skills, AR table, 6 domain
findings, executable verification criteria.
- [improvement-fields.md](improvement-fields.md) — the per-field improvement rollup
(11 cross-domain/delivery/product fields shipped or specced + documentation debt).
- [research-digest.md](research-digest.md) — the web-verified 2024–2026 canon.
- [RUBRIC.md](RUBRIC.md) — the AR rubric as applied here.
## 6. Recommended follow-up PRs (in leverage order)
1. **Loop-cap sweep** — one-sentence caps in scrum-master, jira-expert, code-to-prd,
research-summarizer (~4 skills → HARNESS-READY).
2. **Bind the gates** — make spec-to-repo's validator and code-to-prd's goldens
*required*; name the bridge in scrum-master/senior-pm SKILL.mds.
3. **AI-evals tool** (F–h) — eval-spec linter + kappa calculator; the single
most-demanded missing PM competency.
4. **Path-B completion** — meeting-analyzer scripts (its spec is deterministic math),
team-communications linter, product-discovery references.
5. **Documentation truth** — product-team README (3 conflicting counts, 9 broken paths),
project-management legacy trio, citation back-fill (F9/F10).
6. **Remaining improvement fields** — DORA/EBM/pre-mortem/RACI (delivery); NSM/PLG
bands/WSJF/ODI/taxonomy linter (product), per the specs in improvement-fields.md.
audit/pm-product-agentic-2026-07/RUBRIC.md
+30
View File
@@ -0,0 +1,30 @@
# Agentic-Readiness Rubric (AR v1) — as applied to product-team + project-management
Audit date: 2026-07-03 · Branch: `claude/pm-audit-agentic-loops-jxurlq`
This audit applies the **same AR v1 rubric** established by the engineering agentic audit
([../engineering-agentic-2026-07/RUBRIC.md](../engineering-agentic-2026-07/RUBRIC.md)) —
six dimensions scored 0–2, answering: **can an agent pick this skill up with a goal and
drive it to a verified close?**
| # | Dimension | 2 means |
|---|---|---|
| AR1 | Goal intake | Forcing questions / intake tool / refuses vague input (exit-code gate) |
| AR2 | Task decomposition | Explicit planning step or tool whose output the workflow consumes |
| AR3 | Deterministic execution | Exact runnable CLIs; output consumed by a named next step |
| AR4 | Verification | Machine-checkable gate the workflow REQUIRES before proceeding |
| AR5 | Loop discipline | Iteration caps, stop conditions, escalation thresholds |
| AR6 | Close-out | Definition of done + state persistence or handoff artifact |
Classes: **HARNESS-READY** (total ≥ 9 AND AR4 ≥ 1 AND AR5 ≥ 1) · **LOOP-CAPABLE** (6–8,
or ≥ 9 failing the AR4/AR5 gate) · **TOOL-ONLY** (3–5) · **PROSE-ONLY** (0–2).
Baseline reference: the June 2026 quality audit of these domains lives at
[../newgen-2026-06/product-pm.md](../newgen-2026-06/product-pm.md); this audit scores the
new agentic dimension and ships the domain harness layer that the scores motivated.
Executable enforcement: the regenerated domain manifests
(`engineering/agent-harness/skills/agent-harness/assets/harnesses/{product-team,project-management}.json`)
record per-skill `agentic_signals`; the two new domain orchestrators (`product-skills`,
`pm-skills`) enforce AR1 (routers with exit-code gates), AR4 (delivery gate / OST linter),
and AR5/AR6 (harness budgets + close refusal) at run time.
audit/pm-product-agentic-2026-07/improvement-fields.md
+66
View File
@@ -0,0 +1,66 @@
# Improvement fields — where investment moves each domain most
Research-backed rollup (sources in [research-digest.md](research-digest.md)) of the
fields where the 2024–2026 canon moved past the two domains' current coverage, ordered
by leverage. Fields marked ✅ shipped in this PR; the rest are the follow-up work list,
each with what a deterministic stdlib tool computes.
## Cross-domain (the harness fields)
- **F1 — Loop discipline (AR5)** ✅ *(orchestrator layer)* / open *(per skill)*. Both
domains had zero iteration caps or stop conditions. Shipped: the two orchestrators
carry budgets (3 attempts/task, 12 iterations/goal), named terminal states, and close
refusal. Open: the one-sentence cap pattern ("max N fix-rerun cycles, then escalate")
ported into scrum-master, jira-expert, code-to-prd, research-summarizer — the four
skills one sentence away from HARNESS-READY.
- **F2 — Goal intake (AR1)** ✅ *(routers)* / open *(per skill)*. Both routers refuse
fuzz with exit codes (2/3); grill commands lock decisions before execution. Open:
refuse-on-missing-inputs blocks in the 14 tool-rich skills that still accept anything.
- **F3 — Bind existing gates** — open, cheapest wins: spec-to-repo's validator and
code-to-prd's goldens exist but aren't *required*; scrum-master/senior-pm SKILL.mds
should name the bridge as their data path (one line each).
## Project-management (delivery) fields
| # | Field | Status | Deterministic tool |
|---|---|---|---|
| F4 | Four Kanban flow metrics + SLE + aging-WIP alerts (Kanban Guide 2025) | ✅ `jira_snapshot_bridge.py --to flow` | WIP, throughput, cycle p50/85/95, work-item age, SLE conformance |
| F5 | Monte Carlo probabilistic forecasting (Vacanti; replaces story-point dates) | ✅ `--forecast N` (seeded; refuses < 10 items) | p50/70/85/95 week ranges |
| F6 | Agentic delegation governance (Linear/Rovo model) | ✅ `delivery_loop_gate.py` | owner/reviewer/acceptance/evidence/close rules G1–G6 |
| F–a | DORA 2025 archetypes + AI-amplifier capabilities check | open | four keys from deploy/incident logs → archetype + enabling-capabilities score |
| F–b | EBM (Scrum.org) four Key Value Areas scorecard | open | map existing metrics → KVA coverage, flag empty CV/UV |
| F–c | Pre-mortem processor (Klein) + RAID hygiene linter | open (playbook documents the loops) | cluster failure reasons → owned risk entries; staleness/owner/mitigation lint |
| F–d | Derived project health vs self-reported RAG ("watermelon" diff) | partial (senior-pm dashboard + bridge signals) | composite from schedule variance, aging WIP, scope churn — diffed against RAG |
| F–e | Async-first meeting audit (GitLab canon) | open | calendar export → async-convertibility classes, recoverable hours |
| F–f | Agent-readiness audit of Jira hygiene (Rovo-era) | open | field completeness %, acceptance-criteria presence, stale statuses → delegation-readiness score |
| F–g | RACI validator | open | exactly-one-A, ≥1 R, overload histogram |
## Product-team fields
| # | Field | Status | Deterministic tool |
|---|---|---|---|
| F7 | Continuous-discovery cadence (Torres weekly habit) | ✅ `discovery_cadence_tracker.py` | streak, coverage, outcome linkage, test throughput → health 0–100 |
| F8 | Opportunity Solution Tree structural linting | ✅ `ost_linter.py` | rules O1–O5 (measurable root, needs-not-features, ≥2 solutions, tests, no orphans) |
| F–h | **AI-feature evals as the PRD quality contract** — the single biggest gap per every 2025–2026 source | reference shipped (`ai_product_evals.md`); tool open | eval-spec linter: golden-set floors, rubric pass-criteria, guardrail SLOs; Cohen's kappa on grader agreement |
| F–i | WSJF / cost-of-delay with rank-stability sensitivity (brackets RICE) | reference shipped (`product_operating_model.md`); tool open | CoD/duration ranking; ±1-step perturbation flags rank flips |
| F–j | Opportunity scoring (Ulwick ODI importance–satisfaction) | open | Opp Score per outcome statement from survey CSV |
| F–k | North Star Metric validator + input tree (Amplitude) | open | leading/value/not-vanity checks; input→NSM correlation |
| F–l | PLG funnel benchmark bands (ProductLed/OpenView) | open | stage conversion vs calibrated bands → weakest-stage verdict |
| F–m | Product operating model maturity (Cagan *Transformed*) | open | questionnaire → per-principle 0–100 gap list |
| F–n | Event-taxonomy / tracking-plan linter (PostHog-era) | open | snake_case, verb allowlist, near-duplicate detection |
| F–o | JTBD switch-interview force coder (Moesta) | open | four-forces lexicon coding, force balance per interview |
| F–p | Story-map validator (Patton) | open | every story on the backbone; slices span end-to-end |
| F–q | Model-card completeness checker (Mitchell et al.) | open | nine canonical sections → completeness % |
## Documentation-debt fields
- **F9 — Source citations**: 60 of 64 pre-PR reference files across both domains cite
zero sources (six new references cite 6–7 each). Back-fill priority: scrum-master and
senior-pm references (the canon exists — Vacanti, Kanban Guide, DORA, PMBOK).
- **F10 — Counter/path truth**: product-team README (3 conflicting counts, 9 broken
paths), project-management legacy trio (README / IMPLEMENTATION_SUMMARY /
REAL_WORLD_SCENARIO all say "6 skills"), `.codex/instructions.md` broken paths in both
domains. Domain CLAUDE.mds fixed this PR.
- **F11 — Path-B completion**: meeting-analyzer (0 scripts/refs/assets — its own spec is
deterministic math), team-communications (0 scripts, 155 lines of references),
product-discovery (1 reference), 8 asset-less product skills.
audit/pm-product-agentic-2026-07/product-team.md
+98
View File
@@ -0,0 +1,98 @@
# Domain audit: product-team/ — deep audit + agentic readiness
Audited: 2026-07-03 · 17 skills (13 under `skills/` incl. the router, 4 standalone
plugins) · 19 Python tools (all 19 pass `--help`, functional smoke tests on
sample_size_calculator and hig_checker reproduce correct math) · 5 agents + 1 persona ·
8 slash commands · 5 plugins (all `check_plugin_json.py`-clean).
Rubric: [RUBRIC.md](RUBRIC.md). Method: full SKILL.md reads, script smoke tests,
agent/command cross-referencing, counter verification.
## Summary
**Headline finding (now fixed in this PR): no orchestrator, no loop layer.** Before this
PR the domain had zero `context: fork`, zero forcing-question libraries, no `/cs:*`
router/grill commands, and its "router" (`product-skills`, 61 lines) shipped no tools —
the domain predated every v2.8+ convention. Three skills (spec-to-repo, code-to-prd,
research-summarizer) independently invented verification loops; nothing unified them.
**Agentic-readiness distribution (17 skills, post-PR):** HARNESS-READY **2**
(product-skills upgraded 1→12; spec-to-repo) · LOOP-CAPABLE **4** (product-manager-toolkit,
apple-hig-expert, code-to-prd, research-summarizer) · TOOL-ONLY **11** · PROSE-ONLY **0**.
Weakest dimensions domain-wide: **AR5 loop discipline** (only spec-to-repo has any
retry/stop language) and **AR1 goal intake** (14 of 17 accept any input silently).
## Per-skill table
Scores AR1·AR2·AR3·AR4·AR5·AR6 (post-PR where this PR changed the skill).
| Skill | AR1-6 | Tot | Class | Top improvement |
|---|---|---|---|---|
| skills/product-skills (orchestrator) | 2·2·2·2·2·2 | 12 | HR | (upgraded this PR: was a 61-line prose router, PROSE-ONLY) |
| skills/spec-to-repo | 1·2·2·2·1·1 | 9 | HR | Wire to an agent + command (currently orphaned from both) |
| skills/product-manager-toolkit | 1·1·2·1·0·1 | 6 | LC | Make the PRD checklist a blocking gate; add WSJF/CoD lane + eval-spec PRD section |
| code-to-prd (standalone) | 1·2·2·2·0·2 | 9 | LC (AR5 gate) | One sentence: max 2 analyze-fix cycles vs golden `expected_outputs/`, then escalate |
| research-summarizer (standalone) | 1·1·2·2·0·1 | 7 | LC | Cap the Verification Loop (2 re-extraction passes) → instant HR |
| apple-hig-expert (standalone) | 1·0·2·2·0·1 | 6 | LC | Numbered workflow; move `templates/` → `assets/`; add fix-recheck cap |
| skills/experiment-designer | 1·1·2·1·0·0 | 5 | TO | Make sample-size output a blocking gate on any test recommendation; 423 words is thin |
| skills/ux-researcher-designer | 0·1·2·1·0·1 | 5 | TO | (fake `--help` + unseeded RNG fixed this PR) Validation checklists → exit gates |
| skills/ui-design-system | 0·1·2·1·0·1 | 5 | TO | Add command; disambiguate vs markdown-html/design-system |
| skills/saas-scaffolder | 0·1·2·1·0·1 | 5 | TO | Make the 33-item checklist machine-checkable (reuse spec-to-repo's validator); dedupe references pair |
| agile-product-owner (standalone) | 0·1·2·1·0·1 | 5 | TO | (fake `--help` fixed this PR) INVEST checklist → gate |
| skills/product-analytics | 0·1·2·1·0·0 | 4 | TO | NSM validator + benchmark bands (see improvement-fields F7/F8); anti-patterns table exists, gate doesn't |
| skills/landing-page-generator | 0·1·2·1·0·0 | 4 | TO | Add `distinct_from` vs marketing/landing; render-check gate on emitted TSX |
| skills/product-strategist | 0·1·2·0·0·1 | 4 | TO | Alignment score exists but nothing requires it; outcome-vs-output OKR lint |
| skills/competitive-teardown | 0·1·2·0·0·0 | 3 | TO | Data-verification discipline (it synthesizes scraped claims with no source gate) |
| skills/product-discovery | 0·1·2·0·0·0 | 3 | TO | Wire to the new discovery loop (orchestrator now provides tracker + OST linter); 1 reference, no agent/command |
| skills/roadmap-communicator | 0·0·2·1·0·0 | 3 | TO | Merge-or-disambiguate vs `/changelog` + engineering/changelog-generator; 367 words |
## Domain-level findings
1. **Orchestration gap (fixed this PR).** `product-skills` is now a `context: fork`
orchestrator with a deterministic 16-lane router (`product_goal_router.py`, exit
0/2/3), a recurring discovery loop with two machine gates
(`discovery_cadence_tracker.py`, `ost_linter.py`), a forcing-question library, and
agent-harness integration. Agent `cs-product-orchestrator` + `/cs:product`,
`/cs:grill-product`, `/cs:product-loop` commands added.
2. **References cite no sources: 39 of 43 reference files contain zero URLs/citations.**
Only apple-hig-expert (3/3) and research-summarizer (1/2) meet the ≥5-sources bar. The
3 new orchestrator references cite 7 sources each; the other 39 remain open work.
3. **Two tools faked their `--help` (fixed this PR).** `user_story_generator.py` and
`persona_generator.py` exited 0 while ignoring the flag and running demos;
persona_generator was additionally non-deterministic (unseeded `random.choice`). Both
now use argparse; personas are seeded (default 42).
4. **Stale/contradictory counters + broken paths (open).** product-team/README.md holds 3
mutually inconsistent counts and 9 broken Quick Start paths (missing `skills/`
segment); `.codex/instructions.md` has 3 broken paths; CLAUDE.md says "13 skills" then
lists 16, claims 17 tools (actual 19). The domain plugin.json description claims
skills the bundle doesn't contain. CLAUDE.md counters updated this PR; README overhaul
is follow-up F10.
5. **Unmanaged overlaps (open):** landing-page-generator ↔ `marketing/landing`;
saas-scaffolder ↔ spec-to-repo; roadmap-communicator ↔ `/changelog` +
`engineering/changelog-generator`; ui-design-system ↔ `markdown-html/design-system`.
Only research-summarizer ships a "Distinct From" section. The orchestrator's routing
table now provides partial disambiguation; per-skill `distinct_from` notes remain.
6. **8 of 13 bundled skills ship 0 assets** despite the repo's template-heavy principle;
apple-hig-expert uses a nonstandard `templates/` dir.
7. **Agent/command coverage holes (open):** 6 skills map to no cs-* agent
(product-discovery, roadmap-communicator, spec-to-repo, code-to-prd, apple-hig-expert,
research-summarizer); 10 have no slash command. The orchestrator router reaches all 17
lanes, which mitigates but does not close this.
## Verification criteria (executable)
- **product-skills (orchestrator):** `python3 product-team/skills/product-skills/scripts/product_goal_router.py --sample`
exits 0 and routes to `product-discovery`; `--text "hello"` exits 3;
`discovery_cadence_tracker.py --input assets/sample_discovery_log.json` exits 0 with
`health_score` 62.0 and verdict `AT-RISK`; `ost_linter.py --input assets/sample_ost.json`
exits 2 with exactly one O2 and one O4 violation; `--sample` variants all exit 0.
- **spec-to-repo:** `validate_project.py --strict` on a scaffolded repo exits 0 before
the workflow may report done (existing contract, holds).
- **code-to-prd:** analyzer output diffs clean against `expected_outputs/` goldens
(existing contract, holds).
- **persona_generator (fixed):** `--help` prints argparse usage (not a demo); two `json`
runs produce byte-identical output.
- **user_story_generator (fixed):** `--help` prints argparse usage; `sprint 30` still
plans a 30-point sprint (backward-compatible positional).
- **Manifest truth:** `harness_manifest_builder.py --domain product-team --no-timestamp`
produces a diff-clean `product-team.json` with `product-skills` scoring all five
`agentic_signals` true and 3 wired, sample-supporting tools.
audit/pm-product-agentic-2026-07/project-management.md
+102
View File
@@ -0,0 +1,102 @@
# Domain audit: project-management/ — deep audit + agentic readiness
Audited: 2026-07-03 · 9 skills · 12 Python tools pre-PR (all pass `--help`; end-to-end
runs of velocity_analyzer and project_health_dashboard reproduce documented fixtures),
15 post-PR · 1 agent pre-PR, 2 post · 3 commands pre-PR (+`/sprint-plan` generic), 6
post · plugin.json valid (`["./skills"]` canonical form).
Rubric: [RUBRIC.md](RUBRIC.md). Method: full SKILL.md reads, script smoke tests, MCP
tool-reference grepping, counter verification.
## Summary
**Headline finding #1 (fixed this PR): the MCP↔analytics gap.** The domain bundles a
live Atlassian Remote MCP (`.mcp.json`) and disciplined tool documentation
(`references/atlassian-mcp-tools.md`, verified live 2026-06-10), yet its two analytics
skills (senior-pm, scrum-master) had **zero** MCP references — nothing connected
`searchJiraIssuesUsingJql` output to the scripts' input schemas. Every sprint-health or
velocity run required hand-built JSON. `jira_snapshot_bridge.py` closes this: raw MCP
search results → scrum-master sprint schema (verified: piped output runs
velocity_analyzer clean) → plus the four Kanban flow metrics + seeded Monte Carlo
forecasting the domain never had.
**Headline finding #2 (fixed this PR): pre-modern agentics.** Zero `context: fork`, zero
forcing questions, no `/cs:*` namespace, no loop with named terminal states. What existed
was raw material: atlassian-admin's 7 concrete VERIFY steps, scrum-master's
data-sufficiency gates with pinned expected outputs (avg 20.2 pts, health 78.3,
action-item completion 46.7%), confluence/templates' verify-before-proceed steps.
**Agentic-readiness distribution (9 skills, post-PR):** HARNESS-READY **1** (pm-skills
upgraded 1→12) · LOOP-CAPABLE **4** (scrum-master, jira-expert, atlassian-admin,
atlassian-templates) · TOOL-ONLY **3** (senior-pm, confluence-expert, meeting-analyzer) ·
PROSE-ONLY **1** (team-communications).
## Per-skill table
Scores AR1·AR2·AR3·AR4·AR5·AR6 (post-PR where this PR changed the skill).
| Skill | AR1-6 | Tot | Class | Top improvement |
|---|---|---|---|---|
| pm-skills (orchestrator) | 2·2·2·2·2·2 | 12 | HR | (upgraded this PR: was a 50-line prose router, PROSE-ONLY) |
| scrum-master | 1·1·2·2·0·1 | 7 | LC | One sentence: cap re-analysis at 2 passes then escalate → instant HR; consume the bridge (`--to sprint`) instead of hand-built JSON |
| atlassian-admin | 0·1·2·2·0·2 | 7 | LC | Intake gate (refuse without approver named); its VERIFY steps are the domain's best — port the pattern to siblings |
| jira-expert | 0·1·2·2·0·1 | 6 | LC | Cap fix-revalidate cycles at 3; ship a sample workflow JSON asset (users must guess the validator's schema) |
| atlassian-templates | 0·1·2·1·1·1 | 6 | LC | Ship static template assets; expected-output fixture for the scaffolder |
| senior-pm | 0·1·2·1·0·1 | 5 | TO | Consume the bridge's flow output in the health dashboard; make KPI thresholds (on-time > 80% etc.) exit-code gates; portfolio-kpis.md is 32 lines |
| confluence-expert | 0·1·2·1·0·1 | 5 | TO | Make its Verify steps blocking; sample input for content_audit_analyzer |
| meeting-analyzer | 1·1·0·1·0·1 | 4 | TO | Ship the deterministic tools its own prose describes (speaking-ratio, filler counts = exactly the repo's "algorithm over AI" case); 0 scripts/refs/assets |
| team-communications | 1·1·0·0·0·0 | 2 | PO | References are 15–65 lines (the skill's premise is "follow the reference exactly"); add a 3P-format linter script |
## Domain-level findings
1. **Orchestration + loop layer (fixed this PR).** `pm-skills` is now a `context: fork`
orchestrator: deterministic 8-lane router (`pm_goal_router.py`), the Jira bridge, and
a delegation-governance gate (`delivery_loop_gate.py` — G1 human owner, G2 reviewer
for agent tasks, G3 machine-checkable acceptance, G4 evidence-before-done, G5 close
refusal, G6 exhausted-budget-is-escalation), all wired to the repo harness
(`assets/harnesses/project-management.json`). Agent `cs-pm-orchestrator` +
`/cs:pm`, `/cs:grill-pm`, `/cs:pm-loop` added. Five reusable PM loops documented in
`references/pm_loop_playbook.md` (sprint-flow, health, retro-action, RAID-hygiene,
comms), each with machine gates and named terminal states.
2. **References cited zero sources (partially fixed).** 0 URLs across all 21 pre-PR
reference files — Schwaber/Sutherland, Vacanti, DORA, Kanban Guide all absent. The 3
new orchestrator references cite 6–7 sources each; back-filling the other 21 is
follow-up F9.
3. **Stale counters everywhere except plugin.json (open).** README ("6 world-class
skills"), IMPLEMENTATION_SUMMARY ("All 6", references `/mnt/user-data/outputs/` build
paths), REAL_WORLD_SCENARIO ("6 Expert Skills"), cs-project-manager agent ("six
skills"), CLAUDE.md (lists 6 of 9 — meeting-analyzer, team-communications, pm-skills
absent). CLAUDE.md updated this PR; the legacy trio (README /
IMPLEMENTATION_SUMMARY / REAL_WORLD_SCENARIO) should be rewritten or retired (F10).
4. **MCP integration is bimodal (structural, now bridged).** Concrete in 4 skills
(jira-expert 14 refs, confluence-expert 11, atlassian-templates 11, atlassian-admin 4
read-only-correct); zero in the 2 analytics skills. The bridge closes the data path;
the two SKILL.mds should still name it (one line each, F3).
5. **Two contributed skills violate the Path-B contract (open).** meeting-analyzer: zero
scripts/references/assets — its own spec (speaking-time %, filler-word counts) is
deterministic computation the repo mandates be scripted. team-communications: zero
scripts, 4 references totaling 155 lines.
6. **`/sprint-plan` counts against product-team but lives half in this domain** — the
sprint-planning integration pattern in CLAUDE.md calls product-team's
user_story_generator; fine, but the CLAUDE.md example used the old positional CLI
(still works — verified backward-compatible after this PR's argparse fix).
## Verification criteria (executable)
- **pm-skills (orchestrator):** `pm_goal_router.py --sample` exits 0 routing to
`scrum-master`; `--text "audit our jira permissions"` exits 2 (single signal → ask);
`--text "hello world"` exits 3. `jira_snapshot_bridge.py --input
assets/sample_jira_snapshot.json --to flow --forecast 20` exits 0 and matches
`assets/expected_flow_metrics.json` (p50=9, p85=14, p95=16 days; 90.9% SLE conformance;
aging alert on PHX-112; forecast p85 = 10 weeks, sampled over zero-filled observed
weeks); `--to sprint` output runs
`velocity_analyzer.py` to exit 0 (avg 11.8 pts over 4 sprints); a 2-sprint snapshot
exits 5. `delivery_loop_gate.py --sample` exits 0; sample plan passes `--mode plan`
(exit 0) and is refused by `--mode close` (exit 4, T2 in_progress).
- **scrum-master:** existing fixture contract holds — velocity_analyzer on
`assets/sample_sprint_data.json` reports avg 20.2 pts on 6 sprints.
- **atlassian-admin:** each VERIFY step names a concrete check (e.g. `GET
/rest/api/3/user?accountId=... returns "active": false`) — keep as the domain's AR4
exemplar.
- **Manifest truth:** `harness_manifest_builder.py --domain project-management
--no-timestamp` produces a diff-clean `project-management.json` with `pm-skills`
scoring all five `agentic_signals` true and 3 wired, sample-supporting tools.
audit/pm-product-agentic-2026-07/research-digest.md
+76
View File
@@ -0,0 +1,76 @@
# Research digest — the 2024–2026 canon behind this audit
Web-verified 2026-07-03. Full citations inline; this digest is the source layer for
[improvement-fields.md](improvement-fields.md) and the six new reference docs shipped
into the two orchestrators.
## Product management: where the canon moved
1. **Discovery became a weekly operating rhythm.** Torres (*Continuous Discovery
Habits*; producttalk.org/opportunity-solution-trees) reframed discovery as weekly
customer touchpoints anchored to one outcome, with the OST as the structural artifact
and assumption tests (Bland, *Testing Business Ideas*) as the unit of progress.
2. **The org-level frame is the product operating model.** Cagan's *Transformed* (SVPG,
2024): empowered teams, outcomes over output, innovation over predictability — 20
first principles (svpg.com/the-product-operating-model-an-introduction).
3. **Evals are the new PRD for AI features** — the consensus 2025 AI-PM competency:
golden set + rubric + guardrail SLOs before building (Lenny's Newsletter "Beyond vibe
checks"; Braintrust "Evals for PMs"; Aakash Gupta "AI Evals"); model cards for the
buyer-facing half (Mitchell et al., arxiv.org/abs/1810.03993).
4. **Metrics spine: North Star + input tree** (Amplitude, *The North Star Playbook*),
with PLG benchmark bands for verdicts (ProductLed; OpenView: activation median ~17%,
best-in-class 33–50%+; free→paid median ~9%, PQL-driven 25–30%).
5. **Prioritization is a bracket, not a framework**: RICE (steady state) + WSJF/cost of
delay (Reinertsen; SAFe WSJF + Yip's false-precision critique) + ODI opportunity
scoring (Ulwick). Sensitivity analysis counters the documented WSJF failure mode.
6. **Analytics practice is taxonomy-first** (PostHog product-analytics best practices):
naming discipline and tracking-plan review before any metric above it.
## Project management / delivery: where the canon moved
1. **Flow metrics are mandatory, not optional.** The Kanban Guide (May 2025,
kanbanguides.org) mandates exactly four measures — WIP, throughput, cycle time, work
item age — plus an SLE; age is the leading indicator.
2. **Forecasting went probabilistic.** Vacanti (*Actionable Agile Metrics*; *When Will
It Be Done?*; scrum.org Monte Carlo guidance): sample historical throughput, answer
with p50/70/85/95 ranges, never a date; refuse thin history.
3. **DORA 2025** (dora.dev/dora-report-2025) replaced elite/high/medium/low with seven
team archetypes over eight measures; core finding: AI **amplifies** existing org
strengths/dysfunctions (individual output up ~98% more merged PRs, org delivery flat
without enabling capabilities). SPACE (Forsgren/Storey, ACM Queue) remains the
multi-dimension corrective; EBM (scrum.org) the value-measurement frame.
4. **Risk practice:** Klein's pre-mortem (HBR 2007, ~30% better risk identification);
RAID hygiene as a linting problem; derived health vs self-reported RAG to catch
watermelon projects.
5. **Async-first delivery:** GitLab handbook (handbook.gitlab.com, asynchronous work) —
written 3-question standups 3–5 min vs 15–30 sync; ~37% meeting-hour reduction. Moghe,
*The Async-First Playbook* (2023).
6. **The vendors shipped agentic PM.** Atlassian Rovo GA'd agents in Jira (assignable,
@mentionable, "every action logged and auditable", Teamwork Graph 150B+ connections,
MCP access — atlassian.com/software/rovo; Team '26 coverage, SiliconANGLE 2026-05-06).
Linear shipped the accountability pattern: agent as contributor, **human stays
primary assignee** (linear.app/agents; changelog 2026-03-24).
## Agentic harness design principles (applied in this PR)
1. **Workflows first, agents when needed** — Anthropic, "Building Effective Agents"
(anthropic.com/research/building-effective-agents): prompt chaining, routing,
parallelization, orchestrator-workers, evaluator-optimizer; the last "when there are
clear evaluation criteria and iterative refinement provides measurable value."
→ The routers are workflows; the loops engage only for goals with fresh feedback.
2. **Definition-of-done must be machine-checkable; never trust self-report** — the
plan→act→verify(deterministic)→reflect shape. → G3/G4 in `delivery_loop_gate.py`;
OST linter exit codes; agent-harness's evidence rule.
3. **Budgets and stop conditions are first-class** — max iterations, attempt caps,
escalation on confidence loss; otherwise reflection is infinite retry. → 3/12 caps,
G6, terminal-state taxonomy (loop-library: success, clean no-op, blocked,
approval-required, exhausted, stagnated).
4. **Human accountability stays attached to delegated work** (Linear; Rovo audit
discipline). → G1/G2; no un-reviewed Jira transitions; admin actions are
approval-required states.
5. **Context via structured interfaces, not prompt-stuffing** (Teamwork Graph / MCP;
Anthropic's ACI emphasis). → snapshot-file pattern: every loop iteration is
executable by a fresh session from files.
6. **Evaluator-optimizer pairs with PM-owned evals** — the golden set + rubric IS the
evaluator's criteria; the loop may never edit the gate it is judged by (this repo's
autoresearch locked-evaluator invariant, generalized).