chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"name": "research-ops-skills",
|
||||
"description": "4 Research-Operations skills + 1 orchestrator: clinical-research (study design: protocol synopsis, endpoint selection, sample-size/power, phase-gating, feasibility), research-finance (R&D program budgeting, burn/runway, F&A indirect-rate modeling, capitalize-vs-expense routing, portfolio ROI), market-research (TAM/SAM/SOM both-methods, survey/sampling design, segmentation, CI synthesis), product-research (interview/JTBD/usability/concept-test design, saturation, insight repository synthesis). Orchestrator skill uses context: fork. Each sub-skill ships per-skill onboarding (onboard.py), a customization loader (config_loader.py) consumed by every tool, and an isolated opt-in autoresearch evaluator (ar_evaluator.py) bridging to engineering/autoresearch-agent. 24 stdlib-only Python tools (12 analysis + 12 onboarding/customization/autoresearch), 12 reference docs. Distinct from ra-qm-team (regulatory/QM submission), finance (corporate close/valuation), research/grants (NIH funding discovery), product-team (persona/journey/live experiments), marketing-skill (campaign analytics).",
|
||||
"version": "2.9.0",
|
||||
"author": {
|
||||
"name": "Alireza Rezvani",
|
||||
"url": "https://alirezarezvani.com"
|
||||
},
|
||||
"homepage": "https://github.com/alirezarezvani/claude-skills/tree/main/research-ops",
|
||||
"repository": "https://github.com/alirezarezvani/claude-skills",
|
||||
"license": "MIT",
|
||||
"skills": [
|
||||
"./skills/research-ops-skills",
|
||||
"./skills/clinical-research",
|
||||
"./skills/research-finance",
|
||||
"./skills/market-research",
|
||||
"./skills/product-research"
|
||||
],
|
||||
"source": {
|
||||
"spec": "documentation/implementation/research-ops-expansion-plan.md",
|
||||
"build_pattern": "Path B (direct conversion) — orchestrator skill uses context: fork. Single domain plugin (commercial/ + business-operations/ pattern), NOT the research/ parent-of-plugins pattern. Every SKILL.md ships a Forcing-question library section per Matt Pocock grill-with-docs discipline. Each sub-skill also ships onboard.py + config_loader.py (customization consumed by every tool, project>global>defaults precedence) and an isolated opt-in ar_evaluator.py that bridges to engineering/autoresearch-agent (loop edits the skill's input file; evaluator is locked ground truth).",
|
||||
"distinct_from": "ra-qm-team (ISO 13485/14971, EU MDR, FDA 510(k)/PMA/De Novo/QSR submission — clinical-research designs the prospective study, not the submission). finance/financial-analysis (DCF, ratio analysis, close + report — research-finance manages internal R&D program spend). research/grants (NIH funding DISCOVERY + positioning — research-finance manages money already won). product-team/ux-researcher-designer + product-discovery + experiment-designer (persona/journey artifacts, discovery sprints, live A/B — product-research is method + repository discipline). marketing-skill/campaign-analytics + marketing-demand-acquisition (attribution/ROAS/demand-gen — market-research is upstream methodology)."
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
# Research Operations — Domain Guide
|
||||
|
||||
This file provides domain-specific guidance for skills in `research-ops/`.
|
||||
|
||||
## Purpose
|
||||
|
||||
The Research Operations domain ships skills that help **R&D leads, clinical study teams, R&D finance/controllers, market-research analysts, and product-research / ResearchOps teams** plan, fund, scope, and synthesize research across enterprise workstreams. This is the **enterprise / cross-functional counterpart** to the academic `research/` domain (litreview, grants, patent, syllabus, pulse, dossier, notebooklm).
|
||||
|
||||
It is **not regulatory submission** (`ra-qm-team`), **not corporate financial close/valuation** (`finance/financial-analysis`), **not funding discovery** (`research/grants`), **not persona/journey/live-experiment design** (`product-team`), and **not campaign analytics** (`marketing-skill`).
|
||||
|
||||
## Skills (v2.9.0)
|
||||
|
||||
| Skill | Purpose | `context: fork`? |
|
||||
|---|---|---|
|
||||
| `research-ops-skills` | Domain orchestrator — routes to 4 sub-skills | YES |
|
||||
| `clinical-research` | Study design: protocol synopsis + endpoint selection + sample-size/power + phase-gating | NO |
|
||||
| `research-finance` | R&D program budgeting + burn/runway + F&A rate modeling + capitalize-vs-expense routing | NO |
|
||||
| `market-research` | TAM/SAM/SOM (both methods) + survey/sampling design + segmentation + CI synthesis | NO |
|
||||
| `product-research` | Study design + saturation/sample method + insight repository synthesis | NO |
|
||||
|
||||
## Hard rules (domain-specific)
|
||||
|
||||
1. **clinical-research: outputs are study-design RECOMMENDATIONS signed by a named clinician/biostatistician/regulatory owner.** Power/sample-size is an ESTIMATE with stated assumptions — never presented as clinical fact. Every tool prints an "ESTIMATE — confirm with a biostatistician" banner.
|
||||
2. **research-finance: every budget output surfaces its assumptions block.** Capitalize-vs-expense routes to a NAMED finance owner and never auto-decides accounting treatment.
|
||||
3. **market-research: TAM/SAM/SOM always shows method (top-down AND bottoms-up) + assumptions.** Never a single unsourced number.
|
||||
4. **product-research: never fabricates user insight.** Sample-size/saturation guidance is method-based and surfaces confidence; single-source claims are flagged as anecdotes, not insights.
|
||||
5. **Stdlib-only Python.** Deterministic logic, no LLM calls in scripts.
|
||||
6. **Industry tuning** via `--profile` on every scoring tool.
|
||||
7. **Matt Pocock grill discipline** — `/cs:grill-research-ops` interrogates the plan against the research canon (ICH E9, IAS 38, Cochran, Nielsen, Kotler) before any sub-skill runs.
|
||||
8. **Onboarding-first + customization-in-use.** Each sub-skill ships `scripts/onboard.py` (its own question set) + `scripts/config_loader.py`. Answers persist to `~/.config/research-ops/<skill>.json` (global) or `./.research-ops/<skill>.json` (project) and are consumed by every tool (CLI flags override; `RESEARCH_OPS_NO_CONFIG=1` bypasses). Customization must change behavior, not sit as decoration.
|
||||
9. **Autoresearch is opt-in + isolated.** Each sub-skill ships `scripts/ar_evaluator.py` — a per-skill, locked ground-truth bridge to `engineering/autoresearch-agent`. A loop is invoked ONLY on explicit user request and only edits the skill's input file, never the evaluator. No cross-skill coupling.
|
||||
|
||||
## Build pattern
|
||||
|
||||
Path-B contract per skill: SKILL.md + 3 stdlib scoring scripts + 3 references (each citing 5-7 sources) + 1 asset template, **plus** 3 integration scripts — `onboard.py` (questionnaire), `config_loader.py` (customization loader, project→global→defaults precedence), and `ar_evaluator.py` (isolated autoresearch bridge). SKILL.md includes a "Forcing-question library" section (cited-canon grilling, one question at a time) and the "Onboarding & customization" + "Optimize with autoresearch (opt-in)" sections.
|
||||
|
||||
## Agent + command pattern
|
||||
|
||||
- `cs-research-ops-orchestrator` — evidence-first R&D operations lead. Voice: "What decision does this research drive, and what's your confidence — show me the method and the assumptions before the number."
|
||||
- `/cs:research-ops <inquiry>` — top-level router
|
||||
- `/cs:grill-research-ops <plan>` — Matt-style grilling first
|
||||
- `/cs:clinical-research`, `/cs:research-finance`, `/cs:market-research`, `/cs:product-research` — direct per-skill invocation
|
||||
|
||||
## Anti-patterns (domain-level)
|
||||
|
||||
- ❌ Skills that overlap `ra-qm-team` (regulatory/QM submission) — clinical-research designs the **study**, not the submission
|
||||
- ❌ Skills that overlap `finance/financial-analysis` (close/valuation) — research-finance manages **R&D program spend**
|
||||
- ❌ Skills that overlap `research/grants` (funding discovery) — research-finance manages **money already won**
|
||||
- ❌ Skills that overlap `product-team` (persona/journey/live experiments) — product-research is **method + repository discipline**
|
||||
- ❌ Skills that overlap `marketing-skill` (campaign analytics) — market-research is **upstream methodology**
|
||||
- ❌ A market size stated as a single unsourced number
|
||||
- ❌ A clinical power/endpoint output presented as fact rather than an estimate with a named owner
|
||||
- ❌ A product insight asserted from a single participant
|
||||
|
||||
## References
|
||||
|
||||
- Master plan: `documentation/implementation/research-ops-expansion-plan.md`
|
||||
- Matt Pocock derivation: `engineering/grill-with-docs`
|
||||
- Academic counterpart: `research/` (litreview, grants, patent)
|
||||
- Regulatory complement: `ra-qm-team`
|
||||
- Corporate-finance complement: `finance/financial-analysis`
|
||||
- Product complement: `product-team`
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
name: cs-research-ops-orchestrator
|
||||
description: Evidence-first R&D operations lead. Routes enterprise research inquiries (clinical study design / R&D finance / market research / product research) to the right sub-skill via the research-ops-skills orchestrator. Forks context to keep heavy intake (protocol drafts, program ledgers, survey exports, interview transcripts) out of the parent thread. Signature forcing question — "What decision does this research drive, and what's your confidence?"
|
||||
tools: Read, Write, Edit, Glob, Grep, Bash, Skill
|
||||
model: sonnet
|
||||
---
|
||||
|
||||
# cs-research-ops-orchestrator — Evidence-first R&D operations lead
|
||||
|
||||
You are an enterprise Research Operations lead. You manage **how research is planned, funded, scoped, and synthesized** across four workstreams: clinical R&D, R&D finance, market research, and product research. You are not the regulatory authority, not the corporate CFO, not a grant-finder — you sit between *we-have-a-research-question* and *we-have-a-defensible-answer-with-a-named-owner*.
|
||||
|
||||
## Voice
|
||||
|
||||
Allergic to single unsourced numbers and to outputs presented as fact. You demand the method and the assumptions *before* the number, and you attach a confidence level to everything.
|
||||
|
||||
Your signature opener: **"What decision does this research drive, and what's your confidence — show me the method and the assumptions before the number."**
|
||||
|
||||
The trap you protect against: a vivid anecdote, a top-down "1% of a huge market", a convenience effect size, or a budget with a hidden F&A rate — each presented as if it were settled fact.
|
||||
|
||||
## Your four lanes
|
||||
|
||||
You route every inquiry to one of four sub-skills via the `research-ops-skills` orchestrator (`context: fork`):
|
||||
|
||||
| Lane | Sub-skill | When |
|
||||
|---|---|---|
|
||||
| Clinical | `clinical-research` | Study design, endpoints, sample-size/power, phase-gate feasibility |
|
||||
| R&D finance | `research-finance` | Program budget, burn/runway, capitalize-vs-expense |
|
||||
| Market | `market-research` | TAM/SAM/SOM, survey/sampling, segmentation, CI |
|
||||
| Product | `product-research` | Study method, saturation, insight synthesis |
|
||||
|
||||
## Routing logic
|
||||
|
||||
1. **Detect signals** — keyword classification against the four-lane signal table
|
||||
2. **Score top two** — top ≥ 2 → route confidently
|
||||
3. **Single signal or tie** — one clarifying question with a recommended answer
|
||||
4. **All zero** — ask which of the four lanes applies
|
||||
|
||||
Explore the workspace first: a `protocol.json` → clinical; `program-budget.json` → finance; `tam-model.json` → market; `interview-guide.md` → product. If a filename resolves the lane, route silently.
|
||||
|
||||
## How you communicate (Matt Pocock grill discipline)
|
||||
|
||||
Adopt the five rules from `engineering/grill-with-docs` (Matt Pocock, MIT):
|
||||
|
||||
1. **One question per turn.** Never bundle.
|
||||
2. **Always recommend an answer.** Format: "Recommended: <answer>, because <canon-cited rationale>".
|
||||
3. **Explore before asking.** Check the workspace for protocols, ledgers, market models, interview guides first.
|
||||
4. **Walk the tree depth-first.** Finish a lane before opening another.
|
||||
5. **Track dependencies.** Endpoint → sample size → feasibility; budget → burn → treatment; sizing → survey → segmentation; method → saturation → synthesis.
|
||||
|
||||
After running a sub-skill, return a **≤ 200-word digest**:
|
||||
- What was analyzed
|
||||
- Top 3 findings, each anchored to a canon citation (ICH E9, IAS 38, Cochran, Kotler, Nielsen, etc.)
|
||||
- Top 3 next actions with **named human owner** where applicable
|
||||
- Artifact path
|
||||
- **One grill challenge** for the user, citing canon
|
||||
|
||||
Hard outputs:
|
||||
- Every clinical output is an **estimate** signed by a **named clinical owner** — never clinical fact.
|
||||
- Every finance output surfaces its **assumptions block**; capitalize-vs-expense routes to a **named finance owner**.
|
||||
- Every market size shows **method (both ways) + assumptions** — never a single number.
|
||||
- Every product insight surfaces **confidence + source count**; single-source claims are flagged as anecdotes.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- ❌ Presenting a clinical power/endpoint estimate as fact
|
||||
- ❌ Auto-deciding capitalize-vs-expense instead of routing to a finance owner
|
||||
- ❌ Quoting a TAM as a single unsourced number
|
||||
- ❌ Promoting a single-participant observation to an insight
|
||||
- ❌ Running all 4 sub-skills "to be thorough" — pick one, digest, chain
|
||||
|
||||
## Onboarding-first + autoresearch handoff
|
||||
|
||||
- **Onboarding-first.** When a user starts a fresh research workstream, point them at the relevant sub-skill's `skills/<sub-skill>/scripts/onboard.py` before running its tools. Each skill has its own question set; answers persist to `~/.config/research-ops/<skill>.json` (or `./.research-ops/<skill>.json`) and pre-configure every tool. Treat customization as mandatory discipline — flag it when it's been skipped.
|
||||
- **Autoresearch is opt-in and isolated.** Each sub-skill ships its own `skills/<sub-skill>/scripts/ar_evaluator.py` bridging to `engineering/autoresearch-agent`. Invoke an autoresearch loop ONLY when the user explicitly asks to optimize / improve / run a loop. The connection is per-skill (no shared coupling): the loop edits the skill's input file; the evaluator is locked ground truth (never edited). Metrics: clinical `feasibility_composite` (↑), finance `runway_months` (↑), market `tam_divergence` (↓), product `validated_insights` (↑).
|
||||
|
||||
## When to escalate
|
||||
|
||||
- Regulatory submission (510(k)/PMA/MDR/QMS) → `ra-qm-team`
|
||||
- Grant FUNDING discovery → `research/grants`
|
||||
- Corporate valuation / close / fundraising → `finance/financial-analysis` (or `cs-cfo-advisor`)
|
||||
- Live product A/B experiment → `product-team/experiment-designer`
|
||||
- Persona / journey artifacts → `product-team/ux-researcher-designer`
|
||||
- Live-campaign optimization → `marketing-skill`
|
||||
|
||||
## Available commands
|
||||
|
||||
- `/cs:research-ops <inquiry>` — your top-level router
|
||||
- `/cs:grill-research-ops <plan>` — Matt-style grilling first
|
||||
- `/cs:clinical-research` — direct invocation of clinical-research
|
||||
- `/cs:research-finance` — direct invocation of research-finance
|
||||
- `/cs:market-research` — direct invocation of market-research
|
||||
- `/cs:product-research` — direct invocation of product-research
|
||||
|
||||
Per-skill onboarding: `python3 skills/<skill>/scripts/onboard.py`. Per-skill autoresearch evaluator: `python3 skills/<skill>/scripts/ar_evaluator.py` (used by `/ar:setup` only on explicit opt-in).
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
description: Clinical study design. Select and classify endpoints, estimate sample size / power (means / proportions / survival), and score a study plan for a GO / GO-WITH-CONDITIONS / REDESIGN / NO-GO phase-gate decision. Every output is an ESTIMATE plus a named clinical owner — never clinical fact. Direct invocation of the clinical-research skill.
|
||||
argument-hint: "<study context: indication, design, endpoints, effect size, target enrollment>"
|
||||
---
|
||||
|
||||
# /cs:clinical-research — Endpoint selection + sample-size + phase-gate feasibility
|
||||
|
||||
Run the `clinical-research` skill on this input:
|
||||
|
||||
**$ARGUMENTS**
|
||||
|
||||
## Three-tool workflow
|
||||
|
||||
1. **`endpoint_selector.py`** — Score candidate endpoints across clinical relevance, measurability, regulatory acceptance, sensitivity-to-change, and burden. Classify PRIMARY / KEY-SECONDARY / EXPLORATORY. Flags unvalidated surrogates (cannot be primary). Industry tuning via `--profile`.
|
||||
|
||||
2. **`sample_size_estimator.py`** — Closed-form power / sample size for two-arm means (Cohen's d), proportions (normal approx), or survival (Schoenfeld events). Inflates for dropout. The effect/difference/HR must trace to a published or anchor-based source.
|
||||
|
||||
3. **`phase_gate_scorer.py`** — Score the study plan 0-100 across recruitment feasibility, endpoint readiness, statistical power, operational complexity, and budget fit. Verdict + named owners (PI, Medical Monitor, Biostatistician, Regulatory Owner).
|
||||
|
||||
## Output
|
||||
|
||||
- Endpoint classification + surrogate flags
|
||||
- Sample-size estimate with assumptions block
|
||||
- Phase-gate verdict with named owner chain
|
||||
- Top 3 next actions
|
||||
|
||||
## Hard rule
|
||||
|
||||
**Every output is an ESTIMATE, not a protocol.** A biostatistician, medical monitor, and regulatory owner sign the final design.
|
||||
|
||||
## First run + optimization
|
||||
|
||||
- **Onboard first:** `python3 skills/clinical-research/scripts/onboard.py` (area, alpha, power, dropout, named owners) — saved config pre-configures every tool. `--show` lists the questions.
|
||||
- **Optimize (opt-in):** only if the user asks to optimize/run a loop, hand off to autoresearch via `skills/clinical-research/scripts/ar_evaluator.py` (`feasibility_composite`, higher is better).
|
||||
|
||||
## Distinct from
|
||||
|
||||
- `ra-qm-team` — that's the regulatory **submission**. This designs the **study**.
|
||||
- `research/grants` — that **finds funding**. This **designs the trial**.
|
||||
- `product-team/experiment-designer` — that's a **product A/B**. This is a **clinical trial**.
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
description: Matt Pocock-style docs-anchored grilling for a Research Operations plan — clinical study, R&D budget, market size, or product study. Walks the plan against the research canon (ICH E9, IAS 38, Cochran, Kotler, Nielsen) one question at a time, recommends an answer per question, and refuses to invoke any sub-skill until the lane-defining decisions are locked. Use before running /cs:research-ops on a fuzzy plan.
|
||||
argument-hint: "<plan, study, budget, market question, or fuzzy research problem>"
|
||||
---
|
||||
|
||||
# /cs:grill-research-ops — Research grill against the research-ops canon
|
||||
|
||||
Apply Matt Pocock's `grill-with-docs` discipline to this plan / problem:
|
||||
|
||||
**$ARGUMENTS**
|
||||
|
||||
## Five rules (preserved from Matt Pocock, MIT)
|
||||
|
||||
1. **One question per turn.** Never bundle.
|
||||
2. **Recommend an answer with each question.**
|
||||
3. **Explore the workspace before asking** — protocols, ledgers, market models, interview guides.
|
||||
4. **Walk depth-first.**
|
||||
5. **Track dependencies** — endpoint → power → feasibility; budget → burn → treatment; sizing → survey → segmentation; method → saturation → synthesis.
|
||||
|
||||
## The Research-Ops decision tree (depth-first)
|
||||
|
||||
### Branch 1 — Which lane?
|
||||
|
||||
- CLINICAL / RD_FINANCE / MARKET / PRODUCT
|
||||
|
||||
### Branch 2 — The forcing question per lane
|
||||
|
||||
**CLINICAL:** "Is your primary endpoint a clinical outcome or a surrogate — and if surrogate, is it validated for this indication?"
|
||||
Recommended: clinical outcome unless the surrogate is on FDA's validated table.
|
||||
Canon: FDA Surrogate Endpoint Table; BEST glossary; ICH E9.
|
||||
|
||||
**RD_FINANCE:** "Is this spend in the research phase or the development phase, and can you evidence technical feasibility?"
|
||||
Recommended: research = expense; development = capitalize-candidate only with feasibility evidence, routed to a named finance owner.
|
||||
Canon: IAS 38; ASC 730.
|
||||
|
||||
**MARKET:** "Is your TAM top-down or bottoms-up — and have you computed it both ways to triangulate?"
|
||||
Recommended: both; reconcile the delta before quoting a number.
|
||||
Canon: Bessemer / a16z market-sizing; Fermi estimation.
|
||||
|
||||
**PRODUCT:** "Is this study generative (discover problems) or evaluative (test a solution)?"
|
||||
Recommended: name it first; the method follows.
|
||||
Canon: Rohrer's UX-research methods landscape (NN/g).
|
||||
|
||||
### Branch 3 — Confidence & assumptions check
|
||||
|
||||
"What's your confidence level, and what are the three assumptions the answer rests on?"
|
||||
Recommended: state confidence (high/moderate/low) and surface the assumptions before any number.
|
||||
|
||||
### Branch 4 — Named owner
|
||||
|
||||
"Who is the human owner who signs this output?"
|
||||
Recommended: a named clinician/biostatistician (clinical), a named finance controller (finance), a named decision-maker (market/product) — not "the team".
|
||||
|
||||
### Branch 5 — Now invoke the sub-skill
|
||||
|
||||
Only after branches 1-4 are locked, invoke `/cs:research-ops` with the synthesized inquiry.
|
||||
|
||||
## Output format per turn
|
||||
|
||||
```
|
||||
Q[i]/[total]: [precise question]
|
||||
Recommended: [answer + canon-cited rationale]
|
||||
|
||||
(Confirm, or override?)
|
||||
```
|
||||
|
||||
## Stop conditions
|
||||
|
||||
- All branches resolved → invoke `/cs:research-ops <synthesized>`
|
||||
- User says "stop grilling, just run it" → invoke with whatever's resolved, flag unresolved branches in digest
|
||||
- User abandons → no sub-skill, save partial grill to `research-ops-grill-{timestamp}.md`
|
||||
|
||||
## Distinct from
|
||||
|
||||
- `engineering/grill-me` (Matt Pocock) — generic.
|
||||
- `engineering/grill-with-docs` (Matt Pocock) — codebase + ADR-anchored for engineering. This is **Research-Ops-domain grilling** against the research canon.
|
||||
- `/cs:research-ops` — **executes** routing. This **interrogates** first.
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
description: Market research methodology. Size a market as TAM/SAM/SOM computed BOTH top-down and bottoms-up (never a single number), plan a survey sample size with finite-population correction and per-segment minimums, and score candidate segments against Kotler's criteria. Outputs always show method + assumptions. Direct invocation of the market-research skill.
|
||||
argument-hint: "<market context: total market value, customer count, price, survey params, candidate segments>"
|
||||
---
|
||||
|
||||
# /cs:market-research — TAM/SAM/SOM + survey sampling + segmentation
|
||||
|
||||
Run the `market-research` skill on this input:
|
||||
|
||||
**$ARGUMENTS**
|
||||
|
||||
## Three-tool workflow
|
||||
|
||||
1. **`market_sizer.py`** — Compute TAM/SAM/SOM by BOTH top-down (total market value × fractions) and bottoms-up (customers × price × adoption) methods side-by-side. Reports divergence and flags failed triangulation. Industry tuning via `--profile`. Never returns a single number.
|
||||
|
||||
2. **`sample_size_planner.py`** — Survey sample size from confidence, margin of error, and expected proportion, with the finite-population correction and per-segment minimums (a survey powered overall is not powered per reported segment).
|
||||
|
||||
3. **`segmentation_scorer.py`** — Score candidate segments against Kotler's measurable / substantial / accessible / differentiable / actionable criteria. Enforces a substantiality + accessibility gate; drops demographic slices that are too small or unreachable.
|
||||
|
||||
## Output
|
||||
|
||||
- TAM/SAM/SOM both ways + triangulation flag + assumptions
|
||||
- Survey n (overall + per-segment floors)
|
||||
- Segment scores with TARGET / WATCH / DROP verdicts
|
||||
- Top 3 next actions
|
||||
|
||||
## Hard rule
|
||||
|
||||
**A market size always travels with its method (both ways) and assumptions — never a single unsourced number.**
|
||||
|
||||
## First run + optimization
|
||||
|
||||
- **Onboard first:** `python3 skills/market-research/scripts/onboard.py` (market profile, survey confidence, margin of error, sizing method) — saved config pre-configures every tool. `--show` lists the questions.
|
||||
- **Optimize (opt-in):** only if the user asks to reconcile the sizing/run a loop, hand off to autoresearch via `skills/market-research/scripts/ar_evaluator.py` (`tam_divergence`, lower is better).
|
||||
|
||||
## Distinct from
|
||||
|
||||
- `marketing-skill/campaign-analytics` — that measures a live campaign. This is upstream methodology.
|
||||
- `marketing-skill/marketing-strategy-pmm` — that sets positioning/GTM. This sizes and segments the market.
|
||||
- `commercial/pricing-strategist` — that sets price. This sizes the market.
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
description: Product / user research methodology. Select the right method for the goal (generative vs evaluative vs validation), compute method-based saturation / sample size with an explicit confidence level, and synthesize coded observations into insights while flagging single-source anecdotes. Never fabricates insight. Direct invocation of the product-research skill.
|
||||
argument-hint: "<research context: goal, product stage, segments, coded observations>"
|
||||
---
|
||||
|
||||
# /cs:product-research — Study design + saturation + insight synthesis
|
||||
|
||||
Run the `product-research` skill on this input:
|
||||
|
||||
**$ARGUMENTS**
|
||||
|
||||
## Three-tool workflow
|
||||
|
||||
1. **`study_designer.py`** — Map (research goal × product stage) to an appropriate method and emit a plan skeleton (objective, participant criteria, guide structure, success criteria). Redirects live A/B to `product-team/experiment-designer`.
|
||||
|
||||
2. **`saturation_planner.py`** — Method-based sample guidance with an explicit confidence label: Nielsen problem-discovery (5/segment), Guest et al. thematic saturation (~12), evaluative coverage. Never claims a prevalence rate from a small-n usability test.
|
||||
|
||||
3. **`insight_synthesizer.py`** — Cluster coded observations by tag, count distinct participants, rank by cross-participant recurrence, and flag any candidate below the source threshold as an ANECDOTE — never promoting it to an insight.
|
||||
|
||||
## Output
|
||||
|
||||
- Recommended method + plan skeleton (matched to the goal)
|
||||
- Sample / saturation plan with confidence + limits
|
||||
- Synthesized candidates: INSIGHT vs ANECDOTE with evidence
|
||||
- Top 3 next actions
|
||||
|
||||
## Hard rule
|
||||
|
||||
**Method must match the goal, and an insight requires recurrence across independent participants.** A single quote is an anecdote, not a finding.
|
||||
|
||||
## First run + optimization
|
||||
|
||||
- **Onboard first:** `python3 skills/product-research/scripts/onboard.py` (product profile, insight source-threshold, saturation method, high-stakes flag) — saved config pre-configures every tool. `--show` lists the questions.
|
||||
- **Optimize (opt-in):** only if the user asks to optimize the synthesis/run a loop, hand off to autoresearch via `skills/product-research/scripts/ar_evaluator.py` (`validated_insights`, higher is better).
|
||||
|
||||
## Distinct from
|
||||
|
||||
- `product-team/ux-researcher-designer` — that produces personas/journey artifacts. This is method + repository discipline.
|
||||
- `product-team/product-discovery` — that plans discovery sprints. This designs and synthesizes the research.
|
||||
- `product-team/experiment-designer` — that runs live A/B. This runs qualitative/evaluative research.
|
||||
- `market-research` (sibling) — that studies the market. This studies users.
|
||||
@@ -0,0 +1,39 @@
|
||||
---
|
||||
description: R&D program finance. Build a multi-period program budget with the F&A (indirect) split, track burn rate and runway against value-inflection milestones, and route R&D cost items to a capitalize-vs-expense determination. Every budget surfaces its assumptions; capex-vs-opex routes to a named finance owner and never auto-decides. Direct invocation of the research-finance skill.
|
||||
argument-hint: "<program context: work packages, F&A rate, cash on hand, milestones, cost items>"
|
||||
---
|
||||
|
||||
# /cs:research-finance — Program budget + burn/runway + capex-vs-opex routing
|
||||
|
||||
Run the `research-finance` skill on this input:
|
||||
|
||||
**$ARGUMENTS**
|
||||
|
||||
## Three-tool workflow
|
||||
|
||||
1. **`program_budget_planner.py`** — Build a multi-period budget from work-package lines, apply the F&A rate to an MTDC-style eligible base (excludes capital equipment + subaward portions over $25k), roll up direct / F&A / fully-loaded cost per period with an explicit assumptions block. Industry tuning via `--profile`.
|
||||
|
||||
2. **`burn_runway_tracker.py`** — Compute average + trailing burn, runway in periods/months, and whether each value-inflection milestone is reachable before cash runs out. Flags accelerating burn and below-threshold runway.
|
||||
|
||||
3. **`capex_vs_opex_router.py`** — Score each cost item against IAS 38 development-phase criteria (or flag ASC 730 expense-as-incurred under US GAAP). Route to CAPITALIZE-CANDIDATE / EXPENSE / FINANCE-OWNER-REVIEW with a named owner. Never books an entry.
|
||||
|
||||
## Output
|
||||
|
||||
- Budget rollup (direct / F&A / fully-loaded) with assumptions
|
||||
- Runway + milestone verdicts + flags
|
||||
- Per-item capex/opex routing with named owner
|
||||
- Top 3 next actions
|
||||
|
||||
## Hard rule
|
||||
|
||||
**Every number carries its assumptions; accounting-treatment calls route to a named finance owner.** This skill never books an entry or decides treatment.
|
||||
|
||||
## First run + optimization
|
||||
|
||||
- **Onboard first:** `python3 skills/research-finance/scripts/onboard.py` (R&D area, F&A rate, runway threshold, accounting standard, finance owner) — saved config pre-configures every tool. `--show` lists the questions.
|
||||
- **Optimize (opt-in):** only if the user asks to optimize/extend runway, hand off to autoresearch via `skills/research-finance/scripts/ar_evaluator.py` (`runway_months`, higher is better).
|
||||
|
||||
## Distinct from
|
||||
|
||||
- `finance/financial-analysis` — that's corporate DCF / close / valuation. This is R&D-program-level.
|
||||
- `research/grants` — that **finds funding**. This **manages money already won**.
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
description: Top-level Research Operations router. Classifies an enterprise research inquiry (clinical study design / R&D finance / market research / product research) and forks context to the right sub-skill via the research-ops-skills orchestrator, returning a ≤200-word digest with a named owner and one grill challenge.
|
||||
argument-hint: "<research inquiry: study design, R&D budget, market size, user research, etc.>"
|
||||
---
|
||||
|
||||
# /cs:research-ops — Research Operations router
|
||||
|
||||
Route this inquiry through the `research-ops-skills` orchestrator:
|
||||
|
||||
**$ARGUMENTS**
|
||||
|
||||
## Routing (deterministic, two-signal threshold)
|
||||
|
||||
| Signal class | Keywords | Sub-skill |
|
||||
|---|---|---|
|
||||
| CLINICAL | clinical trial, study design, protocol, endpoint, sample size, power, phase, biostatistics, feasibility | `clinical-research` |
|
||||
| RD_FINANCE | R&D budget, program budget, burn, runway, F&A, indirect rate, capitalize vs expense, portfolio ROI | `research-finance` |
|
||||
| MARKET | TAM, SAM, SOM, market sizing, survey, sampling, margin of error, segmentation, competitive intelligence | `market-research` |
|
||||
| PRODUCT | user interview, JTBD, usability, concept test, discovery research, research repository, insight, saturation | `product-research` |
|
||||
|
||||
1. Explore the workspace first — a resolving filename routes silently.
|
||||
2. Single signal or tie → one clarifying question with a recommended answer.
|
||||
3. Genuine multi-lane → highest-confidence first, run in fork, ask before chaining. Never silently chain.
|
||||
|
||||
## Output (≤200-word digest)
|
||||
|
||||
- What was analyzed
|
||||
- Top 3 findings, each anchored to a canon citation
|
||||
- Top 3 next actions with a named human owner where applicable
|
||||
- Artifact path
|
||||
- One grill challenge for the user
|
||||
|
||||
## Hard rules
|
||||
|
||||
- Clinical output is an estimate + named clinical owner — never fact.
|
||||
- Finance output surfaces assumptions; capex-vs-opex routes to a named finance owner.
|
||||
- Market size shows method (both ways) + assumptions — never a single number.
|
||||
- Product insight surfaces confidence + source count; singletons are anecdotes.
|
||||
|
||||
## Distinct from
|
||||
|
||||
- `research/` (academic) — finds literature/grants/patents. This plans/funds/scopes/synthesizes.
|
||||
- `ra-qm-team` — regulatory submission. `finance/` — corporate close. `marketing-skill` — campaign analytics.
|
||||
@@ -0,0 +1,146 @@
|
||||
---
|
||||
name: clinical-research
|
||||
description: Use when designing a prospective clinical study before submission — selecting and classifying endpoints (primary / key-secondary / exploratory, with surrogate-endpoint flagging), estimating sample size and power for two-arm designs (means / proportions / survival), or scoring a study plan for feasibility and a GO / GO-WITH-CONDITIONS / REDESIGN / NO-GO phase-gate decision. Every output is an ESTIMATE plus a named human owner (clinician / biostatistician / regulatory owner) — never clinical fact, never a finished protocol. Distinct from ra-qm-team, which handles the regulatory/QM submission (ISO 13485, EU MDR, FDA 510(k)/PMA/QSR), not the study design.
|
||||
version: 2.9.0
|
||||
author: claude-code-skills
|
||||
license: MIT
|
||||
tags: [research-ops, clinical-research, study-design, endpoint, sample-size, power, phase-gate, biostatistics]
|
||||
compatible_tools: [claude-code, codex-cli, cursor, antigravity, opencode, gemini-cli]
|
||||
---
|
||||
|
||||
# clinical-research
|
||||
|
||||
Prospective clinical study DESIGN: endpoints, sample size / power, and phase-gate feasibility. Every output is an **estimate with stated assumptions** routed to a **named human owner**. This skill never gives clinical advice as fact and never substitutes for a biostatistician or regulatory affairs.
|
||||
|
||||
## Purpose
|
||||
|
||||
R&D clinical teams, medical monitors, and biostatistics functions live at the moment between *we-have-a-hypothesis* and *we-have-a-protocol-ready-for-submission*. This skill structures three of the hardest design decisions:
|
||||
|
||||
Three deterministic tools:
|
||||
|
||||
1. `sample_size_estimator.py` — Closed-form power / sample-size for two-arm **means** (Cohen's d), **proportions** (normal approximation), and **survival** (Schoenfeld events). Inflates for dropout. Prints an "ESTIMATE — confirm with a biostatistician" banner.
|
||||
2. `endpoint_selector.py` — Scores candidate endpoints across 5 weighted dimensions (clinical relevance, measurability, regulatory acceptance, sensitivity-to-change, burden) and classifies each as **PRIMARY / KEY-SECONDARY / EXPLORATORY**. Penalizes unvalidated surrogate endpoints.
|
||||
3. `phase_gate_scorer.py` — Scores a study plan 0-100 across recruitment feasibility, endpoint readiness, statistical power, operational complexity, and budget fit; returns **GO / GO-WITH-CONDITIONS / REDESIGN / NO-GO** plus the named owners who must sign.
|
||||
|
||||
## When to use
|
||||
|
||||
Invoke this skill when:
|
||||
|
||||
- You are choosing a primary endpoint and need to defend it against surrogate-endpoint scrutiny.
|
||||
- You need a defensible first sample-size estimate for a protocol synopsis.
|
||||
- A study plan needs a feasibility read before a phase-gate review.
|
||||
- You are pressure-testing whether the planned enrollment is achievable given the eligible population and sites.
|
||||
|
||||
**Do NOT use this skill to**: prepare a regulatory submission or clinical evaluation report (use `ra-qm-team`), find or position a grant (use `research/grants`), design a live product A/B experiment (use `product-team/experiment-designer`), or replace a biostatistician's final sample-size justification.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Draft the synopsis** — Fill `assets/protocol_synopsis_template.md` (objectives, design, population, endpoints, statistical plan placeholder, owners-to-sign).
|
||||
2. **Select the endpoint** — Run `endpoint_selector.py --input endpoints.json --profile {drug|device|biologic|diagnostic|digital-therapeutic}`. Read the classification + surrogate flags. If >1 primary, plan multiplicity control.
|
||||
3. **Estimate the sample size** — Run `sample_size_estimator.py --design {means|proportions|survival} ...`. Trace the effect/difference/HR to a published or anchor-based source; inflate for dropout.
|
||||
4. **Score feasibility** — Run `phase_gate_scorer.py --input study.json --profile <same> --phase {1|2|3|4}`. Read the verdict + blockers + named owners.
|
||||
5. **Route for sign-off** — Assemble the synopsis + estimates into the gate packet. The packet is **a recommendation**; a biostatistician, medical monitor, and regulatory owner sign.
|
||||
|
||||
## Scripts
|
||||
|
||||
| Script | Purpose | Profiles |
|
||||
|---|---|---|
|
||||
| `scripts/sample_size_estimator.py` | Power / sample-size for means, proportions, survival | n/a (design-driven) |
|
||||
| `scripts/endpoint_selector.py` | 5-dimension endpoint scoring + classification + surrogate flag | drug, device, biologic, diagnostic, digital-therapeutic |
|
||||
| `scripts/phase_gate_scorer.py` | Feasibility 0-100 + GO/GO-WITH-CONDITIONS/REDESIGN/NO-GO + owners | drug, device, biologic, diagnostic, digital-therapeutic |
|
||||
|
||||
All three: stdlib-only, `--help`, `--sample`, `--output {human,json}`.
|
||||
|
||||
## Onboarding & customization
|
||||
|
||||
Run the onboarding questionnaire **once before you start** — it captures your defaults and named owners so every tool in this skill is pre-configured. Customization is the point: the answers actually change tool behavior.
|
||||
|
||||
```bash
|
||||
python3 scripts/onboard.py # interactive (also: --defaults, --set key=value, --reset)
|
||||
python3 scripts/onboard.py --show # see the questions + current effective config
|
||||
```
|
||||
|
||||
Answers are saved to `~/.config/research-ops/clinical-research.json` (global) or `./.research-ops/clinical-research.json` (`--scope project`) and are read automatically by `config_loader.py`. They set the default development-area **profile**, default **alpha / power / dropout**, and the named **biostatistician / medical monitor / regulatory owner** printed on outputs. CLI flags always override saved config; `RESEARCH_OPS_NO_CONFIG=1` ignores it entirely.
|
||||
|
||||
**The seven questions:** development area · alpha · power · dropout · biostatistician · medical monitor · regulatory owner.
|
||||
|
||||
## Optimize with autoresearch (opt-in)
|
||||
|
||||
This skill ships an **isolated, opt-in** bridge to `engineering/autoresearch-agent`. Only when you ask to "optimize" / "run a loop" does an autoresearch experiment iteratively improve a study plan against this skill's own feasibility score. `scripts/ar_evaluator.py` is the ground-truth evaluator; it prints `feasibility_composite: <0-100>` (higher is better).
|
||||
|
||||
```bash
|
||||
/ar:setup --domain custom --name trial-feasibility \
|
||||
--target study.json \
|
||||
--eval "python3 ar_evaluator.py --target study.json" \
|
||||
--metric feasibility_composite --direction higher
|
||||
/ar:loop custom/trial-feasibility
|
||||
```
|
||||
|
||||
Isolated: no hard dependency — autoresearch runs only on demand, and the loop edits `study.json`, never the evaluator (locked ground truth).
|
||||
|
||||
## References
|
||||
|
||||
- `references/study_design_canon.md` — ICH E8(R1) general considerations; ICH E9 + E9(R1) estimand addendum; CONSORT 2010; SPIRIT 2013; FDA Multiple Endpoints guidance (2022).
|
||||
- `references/endpoint_and_power.md` — Cohen *Statistical Power Analysis*; Schoenfeld (1983) survival sample size; FDA Surrogate Endpoint Table / BEST glossary; FDA PRO guidance (2009); Chow, Shao & Wang *Sample Size Calculations in Clinical Research*.
|
||||
- `references/trial_operations.md` — ICH E6(R2/R3) GCP; TransCelerate risk-based monitoring; FDA RBM guidance; CTTI recruitment best practices; site-feasibility scoring literature.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- Sample-size formulas use normal approximations with a built-in z-table. They are first-pass **estimates**; a biostatistician produces the final justification (and may use simulation, adaptive designs, or exact methods).
|
||||
- The endpoint scorer applies *customary* regulatory priors per development area via `--profile`. Company- or indication-specific precedent overrides the prior.
|
||||
- The phase-gate scorer bakes in a profile cost-per-patient benchmark; pass a real budget to override the default.
|
||||
- An unvalidated surrogate cannot anchor a PRIMARY endpoint — the scorer enforces this with a penalty.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Presenting a power estimate as fact.** Every output is an estimate with a named owner who must sign.
|
||||
- **Powering for a convenience effect size.** The effect must trace to a published or anchor-based MCID, not to the n you can afford.
|
||||
- **Anchoring a primary on an unvalidated surrogate.** Surrogate endpoints need validation evidence for the indication.
|
||||
- **Ignoring multiplicity.** More than one primary endpoint requires pre-specified alpha allocation.
|
||||
- **Skipping dropout inflation.** Raw n undersizes the study; inflate by 1/(1 − dropout).
|
||||
|
||||
## Distinct from
|
||||
|
||||
| Sibling / neighbor | Scope | Difference |
|
||||
|---|---|---|
|
||||
| `ra-qm-team` | ISO 13485 QMS, ISO 14971 risk, EU MDR tech docs + clinical evaluation, FDA 510(k)/PMA/De Novo/QSR submission | That is the **submission**; clinical-research designs the **study** beforehand |
|
||||
| `research/grants` | NIH funding discovery + positioning | That **finds funding**; this **designs the trial** |
|
||||
| `product-team/experiment-designer` | Live product A/B hypothesis + sample size | That is a **product experiment**; this is a **clinical trial** |
|
||||
| `research-finance` (sibling) | R&D program budget + burn | That **funds** the program; this **scopes** the study |
|
||||
|
||||
## Quick examples
|
||||
|
||||
```bash
|
||||
python3 scripts/sample_size_estimator.py --sample
|
||||
python3 scripts/sample_size_estimator.py --design proportions --p1 0.30 --p2 0.45 --dropout 0.15
|
||||
python3 scripts/endpoint_selector.py --sample
|
||||
python3 scripts/phase_gate_scorer.py --sample --output json
|
||||
```
|
||||
|
||||
The sample correctly flags an unvalidated serum-cytokine surrogate (cannot be primary) and ranks PASI-75 as the PRIMARY endpoint; the phase-gate sample returns a verdict with a named owner chain.
|
||||
|
||||
## Forcing-question library (Matt Pocock grill discipline)
|
||||
|
||||
Walked one at a time by `/cs:grill-research-ops` or the orchestrator. Recommended answer + canon citation per question. Never bundled.
|
||||
|
||||
1. **"Is your primary endpoint a clinical outcome or a surrogate — and if surrogate, is it on FDA's validated table?"**
|
||||
Recommended: clinical outcome unless the surrogate is validated for this indication.
|
||||
Canon: FDA Surrogate Endpoint Table; BEST (Biomarkers, EndpointS, and other Tools) glossary.
|
||||
|
||||
2. **"What's the minimal clinically important difference you're powering for — and where did that number come from?"**
|
||||
Recommended: a published or anchor-based MCID, cited; never a convenience effect size.
|
||||
Canon: ICH E9; Cohen *Statistical Power Analysis*.
|
||||
|
||||
3. **"What dropout rate are you assuming, and is the sample size inflated for it?"**
|
||||
Recommended: inflate n by 1/(1 − dropout) using a justified rate.
|
||||
Canon: Chow, Shao & Wang; ICH E9(R1).
|
||||
|
||||
4. **"Single primary endpoint or multiple — and if multiple, what's the multiplicity control?"**
|
||||
Recommended: pre-specify alpha allocation (hierarchical / Bonferroni).
|
||||
Canon: FDA Multiple Endpoints guidance (2022).
|
||||
|
||||
5. **"Who is the named biostatistician / medical monitor / regulatory owner signing this synopsis?"**
|
||||
Recommended: name them now — this output is a recommendation, not a protocol.
|
||||
Canon: ICH E6(R2) GCP roles & responsibilities.
|
||||
|
||||
Walk depth-first. Lock 1-2 before opening 3-5. After all are answered, invoke `endpoint_selector.py` → `sample_size_estimator.py` → `phase_gate_scorer.py`.
|
||||
@@ -0,0 +1,57 @@
|
||||
# Protocol Synopsis — Template
|
||||
|
||||
> Fill this before running the tools. This is a synopsis, not a full protocol. Every section
|
||||
> ends in a named owner who must sign. Output of this skill is an ESTIMATE — a biostatistician,
|
||||
> medical monitor, and regulatory owner sign the final protocol.
|
||||
|
||||
## 1. Study identification
|
||||
- Study ID:
|
||||
- Sponsor / department:
|
||||
- Phase: [1 | 2 | 3 | 4]
|
||||
- Development area / profile: [drug | device | biologic | diagnostic | digital-therapeutic]
|
||||
|
||||
## 2. Objectives
|
||||
- Primary objective:
|
||||
- Secondary objective(s):
|
||||
- Estimand (population, treatment, endpoint, intercurrent-event strategy, summary measure):
|
||||
|
||||
## 3. Design
|
||||
- Type: [parallel-group RCT | crossover | adaptive | single-arm]
|
||||
- Arms & allocation ratio:
|
||||
- Randomization & stratification factors:
|
||||
- Blinding:
|
||||
|
||||
## 4. Population
|
||||
- Indication:
|
||||
- Key inclusion criteria:
|
||||
- Key exclusion criteria:
|
||||
- Estimated eligible population:
|
||||
- Number of sites / countries:
|
||||
|
||||
## 5. Endpoints
|
||||
| Endpoint | Type (clinical / surrogate / PRO) | Validated? | Proposed class (PRIMARY / KEY-SECONDARY / EXPLORATORY) |
|
||||
|---|---|---|---|
|
||||
| | | | |
|
||||
|
||||
- Multiplicity control (if >1 primary):
|
||||
|
||||
## 6. Statistical plan (placeholder — biostatistician owns the final)
|
||||
- Design for sample size: [means | proportions | survival]
|
||||
- Assumed effect size / difference / HR + **source citation**:
|
||||
- alpha (two-sided): ___ power: ___ dropout: ___
|
||||
- Estimated n (from `sample_size_estimator.py`):
|
||||
|
||||
## 7. Feasibility & budget
|
||||
- Target enrollment / enrollment months:
|
||||
- Visits per patient / invasive procedures:
|
||||
- Planned budget (USD):
|
||||
- Phase-gate verdict (from `phase_gate_scorer.py`):
|
||||
|
||||
## 8. Owners to sign (named, not roles)
|
||||
- Principal Investigator:
|
||||
- Medical Monitor:
|
||||
- Biostatistician:
|
||||
- Regulatory Owner:
|
||||
|
||||
## 9. Assumptions register
|
||||
- (List every assumption behind the effect size, dropout, eligible pool, and budget. Each must trace to a source or be flagged as an unverified planning assumption.)
|
||||
@@ -0,0 +1,37 @@
|
||||
# Endpoints and Statistical Power
|
||||
|
||||
Reference for endpoint selection and sample-size estimation. Pairs with `endpoint_selector.py` and `sample_size_estimator.py`.
|
||||
|
||||
## Endpoint hierarchy
|
||||
|
||||
- **Clinical outcome** — directly measures how a patient feels, functions, or survives (mortality, stroke, symptom resolution). Strongest regulatory standing.
|
||||
- **Surrogate endpoint** — a biomarker intended to substitute for a clinical outcome (LDL cholesterol, viral load, tumor response). Only acceptable if **validated** for the specific indication. FDA maintains a public Surrogate Endpoint Table listing surrogates that have supported approvals; the BEST glossary defines the validation hierarchy (candidate → reasonably likely → validated).
|
||||
- **Patient-reported outcome (PRO)** — measured directly from the patient via a validated instrument. FDA's 2009 PRO guidance sets the bar for instrument validity, reliability, and content validity.
|
||||
|
||||
The tool penalizes an **unvalidated surrogate** so it cannot anchor a PRIMARY endpoint — this mirrors the regulatory reality that an unvalidated surrogate carries approval risk.
|
||||
|
||||
## Choosing the effect size (the hardest input)
|
||||
|
||||
The single most consequential — and most abused — input is the assumed effect size. It must be **clinically meaningful** and **externally justified**, never reverse-engineered from the n you can afford.
|
||||
|
||||
- For **means**, the effect is Cohen's d (standardized mean difference). Cohen's conventional small/medium/large (0.2 / 0.5 / 0.8) are last resorts, not anchors — prefer a published or anchor-based MCID.
|
||||
- For **proportions**, specify the control and treatment rates from prior data; the absolute difference drives n.
|
||||
- For **survival**, specify the target hazard ratio; required *events* (not patients) drive power via Schoenfeld's approximation, then n follows from the overall event probability.
|
||||
|
||||
## Power formulas the tool implements
|
||||
|
||||
- **Two-sample means:** n_per_arm = 2·((z_α + z_β)/d)², adjusted for allocation ratio k.
|
||||
- **Two-sample proportions:** n = [z_α·√(2·p̄·q̄) + z_β·√(p₁q₁ + p₂q₂)]² / (p₁ − p₂)².
|
||||
- **Survival (Schoenfeld):** required events E = 4·(z_α + z_β)² / (ln HR)²; n = E / P(event).
|
||||
|
||||
All inflate for dropout by 1/(1 − dropout). These are estimates; a biostatistician produces the binding justification, possibly via simulation.
|
||||
|
||||
## Sources
|
||||
|
||||
1. Cohen, J., *Statistical Power Analysis for the Behavioral Sciences*, 2nd ed. (1988).
|
||||
2. Schoenfeld, D., *Sample-size formula for the proportional-hazards regression model* — Biometrics 1983;39:499-503.
|
||||
3. Chow, Shao, Wang & Lokhnygina, *Sample Size Calculations in Clinical Research*, 3rd ed. (CRC, 2017).
|
||||
4. FDA, *Surrogate Endpoint Resources for Drug and Biologic Development* (public Surrogate Endpoint Table).
|
||||
5. FDA-NIH BEST (Biomarkers, EndpointS, and other Tools) Resource glossary (2016, updated).
|
||||
6. FDA, *Patient-Reported Outcome Measures: Use in Medical Product Development* (2009).
|
||||
7. Fleming & DeMets, *Surrogate end points in clinical trials: are we being misled?* — Ann Intern Med 1996;125:605-613.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Study Design Canon
|
||||
|
||||
Reference knowledge base for prospective clinical study design. Use this when filling the protocol synopsis and defending design choices at a phase gate.
|
||||
|
||||
## The estimand-first mindset (ICH E9(R1))
|
||||
|
||||
Before choosing an endpoint or a sample size, define the **estimand**: the precise treatment effect the trial will estimate. ICH E9(R1) defines five attributes — population, treatment, endpoint (variable), intercurrent-event handling strategy, and population-level summary. Skipping the estimand is the most common cause of a trial that "succeeds" statistically but answers the wrong question. Intercurrent events (treatment discontinuation, rescue medication, death) must have a pre-specified strategy (treatment-policy, hypothetical, composite, while-on-treatment, principal-stratum).
|
||||
|
||||
## Design selection
|
||||
|
||||
- **Parallel-group RCT** — the default for confirmatory efficacy. Two or more arms, randomized, concurrent controls.
|
||||
- **Crossover** — each subject is their own control; only valid for chronic, stable, reversible conditions with adequate washout.
|
||||
- **Adaptive designs** — pre-planned modifications (sample-size re-estimation, arm dropping, seamless phase 2/3). Powerful but require simulation and regulatory pre-agreement (FDA Adaptive Designs guidance, 2019).
|
||||
- **Single-arm** — only defensible with a well-characterized natural history / external control, common in rare disease and oncology early phases.
|
||||
|
||||
## Randomization & blinding
|
||||
|
||||
Randomization removes selection bias; stratify on strong prognostic factors (and always on site in multicenter trials). Blinding (single / double / triple) removes ascertainment and analysis bias. Document the unblinding plan and the DSMB charter for any interim looks.
|
||||
|
||||
## Multiplicity
|
||||
|
||||
Any trial with more than one primary endpoint, more than two arms, or interim analyses inflates the family-wise type-I error. Pre-specify the control strategy: hierarchical (fixed-sequence) testing, Bonferroni / Holm, or a graphical (Bretz-Maurer) approach. The FDA Multiple Endpoints guidance (2022) is the operative reference.
|
||||
|
||||
## Reporting standards as design checklists
|
||||
|
||||
CONSORT 2010 (parallel-group RCT reporting) and SPIRIT 2013 (protocol content) are reporting standards — but used proactively they are design checklists. If you cannot fill a SPIRIT item, the design has a gap.
|
||||
|
||||
## Sources
|
||||
|
||||
1. ICH E8(R1), *General Considerations for Clinical Studies* (2021) — quality-by-design, fit-for-purpose study design.
|
||||
2. ICH E9, *Statistical Principles for Clinical Trials* (1998) and the **E9(R1) Addendum on Estimands and Sensitivity Analysis** (2019).
|
||||
3. Schulz, Altman & Moher, *CONSORT 2010 Statement* — BMJ 2010;340:c332.
|
||||
4. Chan et al., *SPIRIT 2013 Statement: defining standard protocol items for clinical trials* — Ann Intern Med 2013;158:200-207.
|
||||
5. FDA, *Multiple Endpoints in Clinical Trials: Guidance for Industry* (2022).
|
||||
6. FDA, *Adaptive Designs for Clinical Trials of Drugs and Biologics* (2019).
|
||||
7. Friedman, Furberg, DeMets, *Fundamentals of Clinical Trials*, 5th ed. (Springer, 2015).
|
||||
@@ -0,0 +1,33 @@
|
||||
# Trial Operations and Feasibility
|
||||
|
||||
Reference for study feasibility and the phase-gate decision. Pairs with `phase_gate_scorer.py`.
|
||||
|
||||
## Feasibility is the silent killer
|
||||
|
||||
Most trials that fail do not fail on science — they fail on **enrollment**. A study powered for 240 patients across 18 sites assumes a recruitment rate per site per month that is often optimistic by 2-3×. The feasibility scorer enforces two reality checks: the **eligible pool ratio** (eligible population ÷ target enrollment should comfortably exceed 3×, ideally 10×) and **site capacity** (sites × nominal enroll rate × duration vs target). The "enrolling funnel" loses patients at screening, eligibility, and consent — Lasagna's Law (clinicians overestimate the eligible pool the moment a trial opens) is the operative caution.
|
||||
|
||||
## Good Clinical Practice (GCP)
|
||||
|
||||
ICH E6(R2) — and the in-progress E6(R3) — define the responsibilities of sponsors, investigators, and monitors; informed consent; protocol adherence; and the trial master file. A study design that cannot satisfy GCP roles is not gate-ready. Name the Principal Investigator, Medical Monitor, and Biostatistician before the gate.
|
||||
|
||||
## Risk-based monitoring (RBM)
|
||||
|
||||
Centralized, risk-based monitoring (FDA's 2013 guidance, expanded 2023; TransCelerate's RBM methodology) replaces 100% source-data verification with targeted monitoring of the data and processes that most affect patient safety and data integrity. Building RBM into the design lowers operational complexity (a scored dimension).
|
||||
|
||||
## Operational complexity drivers
|
||||
|
||||
Visits per patient, invasive procedures, central-lab logistics, imaging adjudication, and the number of countries all raise operational complexity and recruitment difficulty. The scorer inverts complexity (simpler design → higher score) because every added visit or procedure raises dropout and cost-per-patient.
|
||||
|
||||
## Budget reality
|
||||
|
||||
Cost-per-patient varies enormously by area (a digital-therapeutic at ~$6k/patient vs a biologic at ~$50k+/patient). The scorer compares planned budget to a profile benchmark and flags under-funding below 75% of benchmark. Research-finance (the sibling skill) owns the full program budget; this scorer only checks gate-level adequacy.
|
||||
|
||||
## Sources
|
||||
|
||||
1. ICH E6(R2), *Good Clinical Practice* (2016); ICH E6(R3) draft (2023).
|
||||
2. FDA, *A Risk-Based Approach to Monitoring of Clinical Investigations* (2013; Q&A revision 2023).
|
||||
3. TransCelerate BioPharma, *Risk-Based Monitoring Methodology* position papers.
|
||||
4. CTTI (Clinical Trials Transformation Initiative), *Recruitment* and *Feasibility* recommendations.
|
||||
5. Lasagna, L. — "Lasagna's Law" on the overestimation of eligible patients (clinical-trials folklore widely cited in feasibility literature).
|
||||
6. Treweek et al., *Strategies to improve recruitment to randomised trials* — Cochrane Database Syst Rev 2018.
|
||||
7. Getz & Campo, *Trial complexity and protocol design* — Tufts CSDD impact reports.
|
||||
@@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env python3
|
||||
"""ar_evaluator.py - Autoresearch evaluator for the clinical-research skill (OPT-IN).
|
||||
|
||||
Stdlib-only. This is the ISOLATED bridge to engineering/autoresearch-agent. It does
|
||||
NOT call autoresearch; it is the ground-truth evaluator that an autoresearch loop runs
|
||||
after editing the target study plan. It reads a study-plan JSON (the file the loop
|
||||
optimizes), scores it with phase_gate_scorer, and prints ONE metric line to stdout:
|
||||
|
||||
feasibility_composite: <0-100> (higher is better)
|
||||
|
||||
Usage inside autoresearch (the user opts in explicitly):
|
||||
/ar:setup --domain custom --name trial-feasibility \\
|
||||
--target study.json --eval "python3 ar_evaluator.py --target study.json" \\
|
||||
--metric feasibility_composite --direction higher
|
||||
|
||||
Direct use:
|
||||
python3 ar_evaluator.py --sample
|
||||
python3 ar_evaluator.py --target study.json --profile drug
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import config_loader as cfg # noqa: E402
|
||||
import phase_gate_scorer as pgs # noqa: E402
|
||||
|
||||
METRIC = "feasibility_composite"
|
||||
|
||||
|
||||
def evaluate_target(study: dict, profile: str, phase: int) -> float:
|
||||
result = pgs.evaluate(study, profile, phase)
|
||||
return float(result["composite"])
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
c = cfg.load_config()
|
||||
p = argparse.ArgumentParser(description="Autoresearch evaluator: study-plan feasibility composite.")
|
||||
p.add_argument("--target", help="path to study-plan JSON (or env AR_TARGET)")
|
||||
p.add_argument("--profile", default=None, help="overrides onboarding default_profile")
|
||||
p.add_argument("--phase", type=int, default=2, choices=[1, 2, 3, 4])
|
||||
p.add_argument("--sample", action="store_true", help="evaluate the embedded sample plan")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
profile = args.profile or c.get("default_profile", "drug")
|
||||
if args.sample:
|
||||
study = pgs.SAMPLE
|
||||
else:
|
||||
target = args.target or os.environ.get("AR_TARGET")
|
||||
if not target:
|
||||
print("error: provide --target <study.json> or set AR_TARGET", file=sys.stderr)
|
||||
return 2
|
||||
try:
|
||||
with open(target) as f:
|
||||
study = json.load(f)
|
||||
except (OSError, json.JSONDecodeError) as e:
|
||||
# autoresearch treats a crash as DISCARD; emit N/A and non-zero.
|
||||
print(f"{METRIC}: N/A")
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
phase = study.get("phase", args.phase)
|
||||
value = evaluate_target(study, profile, phase)
|
||||
# The single machine-readable metric line autoresearch parses:
|
||||
print(f"{METRIC}: {value}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,123 @@
|
||||
#!/usr/bin/env python3
|
||||
"""config_loader.py - Customization loader for the clinical-research skill.
|
||||
|
||||
Stdlib-only. Importable from the skill's other scripts. Precedence (highest wins):
|
||||
1. Project config: <cwd>/.research-ops/clinical-research.json
|
||||
2. Global config: ~/.config/research-ops/clinical-research.json
|
||||
3. Built-in DEFAULTS
|
||||
|
||||
The onboarding answers (written by onboard.py) live in these files and are read
|
||||
here so every tool in this skill picks up the user's customization automatically.
|
||||
Set RESEARCH_OPS_NO_CONFIG=1 (or pass --no-config to a tool) to ignore saved config.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
SKILL = "clinical-research"
|
||||
|
||||
GLOBAL_CONFIG_DIR = Path.home() / ".config" / "research-ops"
|
||||
GLOBAL_CONFIG_PATH = GLOBAL_CONFIG_DIR / f"{SKILL}.json"
|
||||
PROJECT_CONFIG_DIRNAME = ".research-ops"
|
||||
|
||||
DEFAULTS: dict[str, Any] = {
|
||||
"version": 1,
|
||||
"skill": SKILL,
|
||||
"default_profile": "drug",
|
||||
"default_alpha": 0.05,
|
||||
"default_power": 0.80,
|
||||
"default_dropout": 0.15,
|
||||
"owners": {
|
||||
"biostatistician": None,
|
||||
"medical_monitor": None,
|
||||
"regulatory_owner": None,
|
||||
},
|
||||
"setup_completed_at": None,
|
||||
}
|
||||
|
||||
|
||||
def project_config_path(cwd: Path | None = None) -> Path:
|
||||
cwd = cwd or Path.cwd()
|
||||
return cwd / PROJECT_CONFIG_DIRNAME / f"{SKILL}.json"
|
||||
|
||||
|
||||
def _read_json(path: Path) -> dict[str, Any] | None:
|
||||
try:
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
return data if isinstance(data, dict) else None
|
||||
except (FileNotFoundError, json.JSONDecodeError, OSError):
|
||||
return None
|
||||
|
||||
|
||||
def _deep_merge(base: dict[str, Any], override: dict[str, Any]) -> dict[str, Any]:
|
||||
out = dict(base)
|
||||
for k, v in override.items():
|
||||
if isinstance(v, dict) and isinstance(out.get(k), dict):
|
||||
out[k] = _deep_merge(out[k], v)
|
||||
else:
|
||||
out[k] = v
|
||||
return out
|
||||
|
||||
|
||||
def load_config(cwd: Path | None = None) -> dict[str, Any]:
|
||||
"""Effective config = DEFAULTS <- global <- project. Honors RESEARCH_OPS_NO_CONFIG."""
|
||||
config = dict(DEFAULTS)
|
||||
if os.environ.get("RESEARCH_OPS_NO_CONFIG") == "1":
|
||||
return config
|
||||
global_cfg = _read_json(GLOBAL_CONFIG_PATH)
|
||||
if global_cfg:
|
||||
config = _deep_merge(config, global_cfg)
|
||||
project_cfg = _read_json(project_config_path(cwd))
|
||||
if project_cfg:
|
||||
config = _deep_merge(config, project_cfg)
|
||||
return config
|
||||
|
||||
|
||||
def setup_completed() -> bool:
|
||||
cfg = _read_json(GLOBAL_CONFIG_PATH) or _read_json(project_config_path())
|
||||
return bool(cfg and cfg.get("setup_completed_at"))
|
||||
|
||||
|
||||
def write_config(config: dict[str, Any], scope: str = "global", cwd: Path | None = None) -> Path:
|
||||
if scope == "project":
|
||||
path = project_config_path(cwd)
|
||||
else:
|
||||
path = GLOBAL_CONFIG_PATH
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("w", encoding="utf-8") as f:
|
||||
json.dump(config, f, indent=2, sort_keys=True)
|
||||
return path
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description=f"Inspect {SKILL} customization config.")
|
||||
p.add_argument("--show", action="store_true", help="Print the effective config")
|
||||
p.add_argument("--status", action="store_true", help="Print setup status + paths")
|
||||
p.add_argument("--sample", action="store_true", help="Print the built-in defaults")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
if args.sample:
|
||||
print(json.dumps(DEFAULTS, indent=2, sort_keys=True))
|
||||
elif args.status:
|
||||
print(json.dumps({
|
||||
"skill": SKILL,
|
||||
"global_config_path": str(GLOBAL_CONFIG_PATH),
|
||||
"global_config_exists": GLOBAL_CONFIG_PATH.exists(),
|
||||
"project_config_path": str(project_config_path()),
|
||||
"project_config_exists": project_config_path().exists(),
|
||||
"setup_completed": setup_completed(),
|
||||
}, indent=2))
|
||||
else:
|
||||
print(json.dumps(load_config(), indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,183 @@
|
||||
#!/usr/bin/env python3
|
||||
"""endpoint_selector.py - Score candidate clinical endpoints and classify each.
|
||||
|
||||
Stdlib-only. Deterministic. NO LLM calls. ESTIMATE / decision-support only —
|
||||
endpoint selection must be confirmed by a clinician + biostatistician + regulatory owner.
|
||||
|
||||
Each candidate endpoint is scored 0-100 across 5 weighted dimensions:
|
||||
1. clinical_relevance does it measure benefit patients care about? (weight 0.30)
|
||||
2. measurability validated instrument, low measurement error? (weight 0.20)
|
||||
3. regulatory_acceptance precedent acceptance by FDA/EMA for this indication (weight 0.25)
|
||||
4. sensitivity_to_change can it detect treatment effect in the trial window? (weight 0.15)
|
||||
5. burden patient/site burden (inverted: low burden = high) (weight 0.10)
|
||||
|
||||
Classification:
|
||||
- top composite -> PRIMARY
|
||||
- composite >= 60 -> KEY-SECONDARY
|
||||
- else -> EXPLORATORY
|
||||
Surrogate endpoints flagged when is_surrogate=true and not validated.
|
||||
|
||||
Profiles tune the regulatory-acceptance prior by development area.
|
||||
|
||||
Usage:
|
||||
python3 endpoint_selector.py --sample
|
||||
python3 endpoint_selector.py --input endpoints.json --profile drug
|
||||
python3 endpoint_selector.py --input endpoints.json --output json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
import config_loader as _cfg
|
||||
except ImportError: # pragma: no cover
|
||||
_cfg = None
|
||||
|
||||
BANNER = "ESTIMATE ONLY — endpoint selection must be confirmed by clinician + biostatistician + regulatory owner."
|
||||
|
||||
WEIGHTS = {
|
||||
"clinical_relevance": 0.30,
|
||||
"measurability": 0.20,
|
||||
"regulatory_acceptance": 0.25,
|
||||
"sensitivity_to_change": 0.15,
|
||||
"burden": 0.10,
|
||||
}
|
||||
|
||||
# Per-area regulatory-acceptance multiplier applied to the regulatory_acceptance score.
|
||||
PROFILES = {
|
||||
"drug": 1.00,
|
||||
"device": 0.95,
|
||||
"biologic": 1.00,
|
||||
"diagnostic": 0.90,
|
||||
"digital-therapeutic": 0.80,
|
||||
}
|
||||
|
||||
SAMPLE = {
|
||||
"indication": "moderate-to-severe plaque psoriasis",
|
||||
"endpoints": [
|
||||
{
|
||||
"name": "PASI-75 at week 16",
|
||||
"is_surrogate": False,
|
||||
"validated": True,
|
||||
"scores": {"clinical_relevance": 90, "measurability": 85, "regulatory_acceptance": 95,
|
||||
"sensitivity_to_change": 90, "burden": 80},
|
||||
},
|
||||
{
|
||||
"name": "Serum cytokine level at week 4",
|
||||
"is_surrogate": True,
|
||||
"validated": False,
|
||||
"scores": {"clinical_relevance": 40, "measurability": 90, "regulatory_acceptance": 30,
|
||||
"sensitivity_to_change": 85, "burden": 50},
|
||||
},
|
||||
{
|
||||
"name": "DLQI (quality of life) at week 16",
|
||||
"is_surrogate": False,
|
||||
"validated": True,
|
||||
"scores": {"clinical_relevance": 75, "measurability": 70, "regulatory_acceptance": 70,
|
||||
"sensitivity_to_change": 65, "burden": 75},
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def score_endpoint(ep: dict, profile_mult: float) -> dict:
|
||||
raw = ep.get("scores", {})
|
||||
flags: list[str] = []
|
||||
composite = 0.0
|
||||
breakdown = {}
|
||||
for dim, w in WEIGHTS.items():
|
||||
s = float(raw.get(dim, 0.0))
|
||||
if dim == "regulatory_acceptance":
|
||||
s = min(100.0, s * profile_mult)
|
||||
composite += s * w
|
||||
breakdown[dim] = round(s, 1)
|
||||
if ep.get("is_surrogate") and not ep.get("validated"):
|
||||
flags.append("UNVALIDATED SURROGATE — not on a validated-surrogate table; confirm acceptability")
|
||||
composite *= 0.7 # heavy penalty: unvalidated surrogate cannot anchor a primary endpoint
|
||||
return {
|
||||
"name": ep.get("name", "UNNAMED"),
|
||||
"composite": round(composite, 1),
|
||||
"breakdown": breakdown,
|
||||
"is_surrogate": bool(ep.get("is_surrogate")),
|
||||
"validated": bool(ep.get("validated")),
|
||||
"flags": flags,
|
||||
}
|
||||
|
||||
|
||||
def classify(scored: list[dict]) -> list[dict]:
|
||||
if not scored:
|
||||
return scored
|
||||
ordered = sorted(scored, key=lambda x: x["composite"], reverse=True)
|
||||
top = ordered[0]["composite"]
|
||||
for i, s in enumerate(ordered):
|
||||
if i == 0 and not s["flags"]:
|
||||
s["classification"] = "PRIMARY"
|
||||
elif i == 0 and s["flags"]:
|
||||
s["classification"] = "KEY-SECONDARY (flagged — cannot be primary)"
|
||||
elif s["composite"] >= 60.0:
|
||||
s["classification"] = "KEY-SECONDARY"
|
||||
else:
|
||||
s["classification"] = "EXPLORATORY"
|
||||
return ordered
|
||||
|
||||
|
||||
def evaluate(data: dict, profile: str) -> dict:
|
||||
if profile not in PROFILES:
|
||||
raise ValueError(f"Unknown profile '{profile}'. Choose from {list(PROFILES)}.")
|
||||
mult = PROFILES[profile]
|
||||
scored = [score_endpoint(ep, mult) for ep in data.get("endpoints", [])]
|
||||
scored = classify(scored)
|
||||
return {
|
||||
"indication": data.get("indication", "UNSPECIFIED"),
|
||||
"profile": profile,
|
||||
"endpoints": scored,
|
||||
"note": "Multiplicity control (e.g., hierarchical alpha allocation) required if >1 primary endpoint.",
|
||||
}
|
||||
|
||||
|
||||
def _render_human(result: dict) -> str:
|
||||
lines = [f"!! {BANNER}", "", f"Indication: {result['indication']} (profile: {result['profile']})", ""]
|
||||
for ep in result["endpoints"]:
|
||||
lines.append(f"[{ep['classification']}] {ep['name']} — composite {ep['composite']}/100")
|
||||
for dim, s in ep["breakdown"].items():
|
||||
lines.append(f" {dim:24s} {s}")
|
||||
for f in ep["flags"]:
|
||||
lines.append(f" ! {f}")
|
||||
lines.append("")
|
||||
lines.append(f"note: {result['note']}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description="Score and classify candidate clinical endpoints (ESTIMATE ONLY).")
|
||||
p.add_argument("--input", help="Path to JSON with {indication, endpoints[]}")
|
||||
p.add_argument("--profile", default=None, choices=list(PROFILES),
|
||||
help="overrides onboarding default_profile")
|
||||
p.add_argument("--output", choices=["human", "json"], default="human")
|
||||
p.add_argument("--sample", action="store_true", help="use the embedded sample")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
conf = _cfg.load_config() if _cfg else {}
|
||||
profile = args.profile or conf.get("default_profile", "drug")
|
||||
data = SAMPLE if (args.sample or not args.input) else json.load(open(args.input))
|
||||
try:
|
||||
result = evaluate(data, profile)
|
||||
except ValueError as e:
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
if args.output == "json":
|
||||
result["_banner"] = BANNER
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(_render_human(result))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,144 @@
|
||||
#!/usr/bin/env python3
|
||||
"""onboard.py - Onboarding questionnaire for the clinical-research skill.
|
||||
|
||||
Stdlib-only. Asks the user a short set of questions BEFORE they start designing a
|
||||
study, then writes their answers to a customization config (read by every tool in
|
||||
this skill via config_loader.py). Customization is the point: the answers become the
|
||||
defaults for profile, alpha/power/dropout, and the named owners printed on outputs.
|
||||
|
||||
Modes:
|
||||
--show print the questions + the current effective config, then exit
|
||||
--defaults write the built-in defaults without prompting (non-interactive)
|
||||
--set key=value ... set specific answers non-interactively (repeatable)
|
||||
--reset delete the saved config at the chosen scope
|
||||
--scope {global,project} where to save (default: global = ~/.config/research-ops)
|
||||
|
||||
With no flags and an interactive terminal, it walks the questions one at a time.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as _dt
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import config_loader as cfg # noqa: E402
|
||||
|
||||
# (key, prompt, choices_or_None, caster, default_key_in_DEFAULTS)
|
||||
QUESTIONS = [
|
||||
("default_profile",
|
||||
"1. What development area are you working in?",
|
||||
["drug", "device", "biologic", "diagnostic", "digital-therapeutic"], str, "default_profile"),
|
||||
("default_alpha",
|
||||
"2. Default two-sided significance level (alpha)?",
|
||||
["0.10", "0.05", "0.025", "0.01"], float, "default_alpha"),
|
||||
("default_power",
|
||||
"3. Default target power (1 - beta)?",
|
||||
["0.80", "0.85", "0.90", "0.95"], float, "default_power"),
|
||||
("default_dropout",
|
||||
"4. Default anticipated dropout fraction (for sample-size inflation)?",
|
||||
None, float, "default_dropout"),
|
||||
("owner.biostatistician",
|
||||
"5. Named biostatistician who signs the sample-size justification?",
|
||||
None, str, None),
|
||||
("owner.medical_monitor",
|
||||
"6. Named medical monitor for the study?",
|
||||
None, str, None),
|
||||
("owner.regulatory_owner",
|
||||
"7. Named regulatory owner who signs the gate decision?",
|
||||
None, str, None),
|
||||
]
|
||||
|
||||
|
||||
def _apply(config: dict, key: str, value) -> None:
|
||||
if key.startswith("owner."):
|
||||
config.setdefault("owners", {})[key.split(".", 1)[1]] = value
|
||||
else:
|
||||
config[key] = value
|
||||
|
||||
|
||||
def _print_questions() -> None:
|
||||
print(f"Onboarding questions — {cfg.SKILL}:\n")
|
||||
for _, prompt, choices, _c, _d in QUESTIONS:
|
||||
line = f" {prompt}"
|
||||
if choices:
|
||||
line += f" [{ ' / '.join(choices) }]"
|
||||
print(line)
|
||||
|
||||
|
||||
def run_interactive(config: dict) -> dict:
|
||||
print(f"Onboarding — {cfg.SKILL}. Press Enter to keep the current/default value.\n")
|
||||
for key, prompt, choices, caster, dkey in QUESTIONS:
|
||||
current = config.get("owners", {}).get(key.split(".", 1)[1]) if key.startswith("owner.") \
|
||||
else config.get(key)
|
||||
suffix = f" [{ '/'.join(choices) }]" if choices else ""
|
||||
cur = f" (current: {current})" if current is not None else ""
|
||||
raw = input(f"{prompt}{suffix}{cur}: ").strip()
|
||||
if not raw:
|
||||
continue
|
||||
try:
|
||||
_apply(config, key, caster(raw))
|
||||
except ValueError:
|
||||
print(f" ! invalid value for {key}, keeping current")
|
||||
return config
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description=f"Onboarding for the {cfg.SKILL} skill.")
|
||||
p.add_argument("--show", action="store_true", help="print questions + effective config")
|
||||
p.add_argument("--defaults", action="store_true", help="write built-in defaults, no prompt")
|
||||
p.add_argument("--set", action="append", default=[], metavar="key=value",
|
||||
help="set an answer non-interactively (repeatable)")
|
||||
p.add_argument("--reset", action="store_true", help="delete saved config at the scope")
|
||||
p.add_argument("--scope", choices=["global", "project"], default="global")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
if args.show:
|
||||
_print_questions()
|
||||
print("\nCurrent effective config:")
|
||||
import json
|
||||
print(json.dumps(cfg.load_config(), indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
if args.reset:
|
||||
path = cfg.project_config_path() if args.scope == "project" else cfg.GLOBAL_CONFIG_PATH
|
||||
if path.exists():
|
||||
path.unlink()
|
||||
print(f"removed {path}")
|
||||
else:
|
||||
print(f"no config at {path}")
|
||||
return 0
|
||||
|
||||
config = cfg.load_config()
|
||||
|
||||
if args.set:
|
||||
for item in args.set:
|
||||
if "=" not in item:
|
||||
print(f"error: --set expects key=value, got '{item}'", file=sys.stderr)
|
||||
return 2
|
||||
k, v = item.split("=", 1)
|
||||
# best-effort type coercion for known numeric keys
|
||||
if k in ("default_alpha", "default_power", "default_dropout"):
|
||||
try:
|
||||
v = float(v)
|
||||
except ValueError:
|
||||
pass
|
||||
_apply(config, k, v)
|
||||
elif not args.defaults:
|
||||
if sys.stdin.isatty():
|
||||
config = run_interactive(config)
|
||||
else:
|
||||
print("non-interactive shell: use --defaults or --set key=value. Showing questions:\n")
|
||||
_print_questions()
|
||||
return 0
|
||||
|
||||
config["setup_completed_at"] = _dt.datetime.now(_dt.timezone.utc).isoformat()
|
||||
path = cfg.write_config(config, scope=args.scope)
|
||||
print(f"saved {cfg.SKILL} customization -> {path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,234 @@
|
||||
#!/usr/bin/env python3
|
||||
"""phase_gate_scorer.py - Score a study plan for feasibility and route a phase-gate verdict.
|
||||
|
||||
Stdlib-only. Deterministic. NO LLM calls. ESTIMATE / decision-support only: the verdict
|
||||
names the human owner(s) who must sign — it never authorizes a study on its own.
|
||||
|
||||
Scores a study plan 0-100 across 5 dimensions:
|
||||
1. recruitment_feasibility eligible-population size vs target enrollment + timeline
|
||||
2. endpoint_readiness endpoint validated + instrument in place
|
||||
3. statistical_power is the planned n adequate for the stated effect?
|
||||
4. operational_complexity sites, visits, procedures (inverted: simpler = higher)
|
||||
5. budget_fit planned budget vs profile cost-per-patient benchmark
|
||||
|
||||
Verdict:
|
||||
- composite >= 80 and no blockers -> GO
|
||||
- composite 65-79 -> GO-WITH-CONDITIONS
|
||||
- composite 50-64 or 1 blocker -> REDESIGN
|
||||
- composite < 50 or 2+ blockers -> NO-GO
|
||||
|
||||
Profiles tune the cost-per-patient benchmark and recruitment difficulty.
|
||||
|
||||
Usage:
|
||||
python3 phase_gate_scorer.py --sample
|
||||
python3 phase_gate_scorer.py --input study.json --profile device --phase 2
|
||||
python3 phase_gate_scorer.py --input study.json --output json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
import config_loader as _cfg
|
||||
except ImportError: # pragma: no cover
|
||||
_cfg = None
|
||||
|
||||
BANNER = "ESTIMATE ONLY — a medical monitor + biostatistician + regulatory owner must sign the gate decision."
|
||||
|
||||
WEIGHTS = {
|
||||
"recruitment_feasibility": 0.25,
|
||||
"endpoint_readiness": 0.20,
|
||||
"statistical_power": 0.25,
|
||||
"operational_complexity": 0.15,
|
||||
"budget_fit": 0.15,
|
||||
}
|
||||
|
||||
# cost_per_patient_usd is the profile benchmark used for budget_fit scoring.
|
||||
PROFILES = {
|
||||
"drug": {"cost_per_patient_usd": 41000, "recruit_difficulty": 1.0},
|
||||
"device": {"cost_per_patient_usd": 28000, "recruit_difficulty": 0.9},
|
||||
"biologic": {"cost_per_patient_usd": 52000, "recruit_difficulty": 1.1},
|
||||
"diagnostic": {"cost_per_patient_usd": 12000, "recruit_difficulty": 0.8},
|
||||
"digital-therapeutic": {"cost_per_patient_usd": 6000, "recruit_difficulty": 0.7},
|
||||
}
|
||||
|
||||
OWNERS = {
|
||||
"GO": ["Principal Investigator", "Medical Monitor", "Biostatistician"],
|
||||
"GO-WITH-CONDITIONS": ["Principal Investigator", "Medical Monitor", "Biostatistician", "Regulatory Owner"],
|
||||
"REDESIGN": ["Medical Monitor", "Biostatistician", "Regulatory Owner", "Study Director"],
|
||||
"NO-GO": ["Medical Monitor", "Biostatistician", "Regulatory Owner", "Study Director", "R&D Head"],
|
||||
}
|
||||
|
||||
SAMPLE = {
|
||||
"study_id": "PSO-2026-P2",
|
||||
"phase": 2,
|
||||
"eligible_population": 4200,
|
||||
"target_enrollment": 240,
|
||||
"enrollment_months": 14,
|
||||
"sites": 18,
|
||||
"endpoint_validated": True,
|
||||
"instrument_in_place": True,
|
||||
"planned_n": 240,
|
||||
"required_n": 260,
|
||||
"visits_per_patient": 9,
|
||||
"invasive_procedures": 2,
|
||||
"planned_budget_usd": 8200000,
|
||||
}
|
||||
|
||||
|
||||
def _clamp(x, lo=0.0, hi=100.0):
|
||||
return max(lo, min(hi, x))
|
||||
|
||||
|
||||
def score_plan(study: dict, profile: dict, phase: int) -> dict:
|
||||
blockers: list[str] = []
|
||||
breakdown = {}
|
||||
|
||||
# 1. recruitment feasibility: eligible pop must dwarf target; ~25 enroll/site/yr nominal
|
||||
elig = float(study.get("eligible_population", 0))
|
||||
target = float(study.get("target_enrollment", 1)) or 1
|
||||
months = float(study.get("enrollment_months", 12)) or 12
|
||||
sites = float(study.get("sites", 1)) or 1
|
||||
pool_ratio = elig / target if target else 0
|
||||
nominal_capacity = sites * 25.0 * (months / 12.0) / profile["recruit_difficulty"]
|
||||
capacity_ratio = nominal_capacity / target if target else 0
|
||||
recruit = _clamp(40.0 * min(pool_ratio / 10.0, 1.0) + 60.0 * min(capacity_ratio, 1.0))
|
||||
if pool_ratio < 3.0:
|
||||
blockers.append("recruitment: eligible pool < 3x target enrollment")
|
||||
breakdown["recruitment_feasibility"] = round(recruit, 1)
|
||||
|
||||
# 2. endpoint readiness
|
||||
er = 0.0
|
||||
er += 60.0 if study.get("endpoint_validated") else 0.0
|
||||
er += 40.0 if study.get("instrument_in_place") else 0.0
|
||||
if not study.get("endpoint_validated"):
|
||||
blockers.append("endpoint: primary endpoint not validated")
|
||||
breakdown["endpoint_readiness"] = round(er, 1)
|
||||
|
||||
# 3. statistical power: planned_n vs required_n
|
||||
planned_n = float(study.get("planned_n", 0))
|
||||
required_n = float(study.get("required_n", 0)) or 1
|
||||
ratio = planned_n / required_n if required_n else 0
|
||||
power = _clamp(100.0 * min(ratio, 1.0)) if ratio >= 1.0 else _clamp(100.0 * ratio - (1.0 - ratio) * 40.0)
|
||||
if ratio < 0.9:
|
||||
blockers.append(f"power: planned n ({planned_n:.0f}) < 90% of required n ({required_n:.0f})")
|
||||
breakdown["statistical_power"] = round(power, 1)
|
||||
|
||||
# 4. operational complexity (inverted: more visits/procedures = lower score)
|
||||
visits = float(study.get("visits_per_patient", 6))
|
||||
procs = float(study.get("invasive_procedures", 0))
|
||||
complexity = _clamp(100.0 - (visits - 4) * 6.0 - procs * 10.0)
|
||||
breakdown["operational_complexity"] = round(complexity, 1)
|
||||
|
||||
# 5. budget fit: planned budget vs benchmark cost-per-patient * target
|
||||
benchmark = profile["cost_per_patient_usd"] * target
|
||||
planned_budget = float(study.get("planned_budget_usd", 0))
|
||||
if planned_budget <= 0:
|
||||
budget = 0.0
|
||||
blockers.append("budget: no planned budget provided")
|
||||
else:
|
||||
coverage = planned_budget / benchmark if benchmark else 0
|
||||
# 100 if planned >= benchmark, sliding down if under-funded
|
||||
budget = _clamp(100.0 * min(coverage, 1.0)) if coverage >= 1.0 else _clamp(coverage * 100.0)
|
||||
if coverage < 0.75:
|
||||
blockers.append("budget: planned budget < 75% of benchmark cost")
|
||||
breakdown["budget_fit"] = round(budget, 1)
|
||||
|
||||
composite = sum(breakdown[d] * w for d, w in WEIGHTS.items())
|
||||
verdict = _verdict(composite, blockers)
|
||||
return {
|
||||
"study_id": study.get("study_id", "UNSPECIFIED"),
|
||||
"phase": phase,
|
||||
"composite": round(composite, 1),
|
||||
"verdict": verdict,
|
||||
"named_owners": OWNERS[verdict],
|
||||
"breakdown": breakdown,
|
||||
"blockers": blockers,
|
||||
"benchmark_cost_usd": round(benchmark, 0),
|
||||
}
|
||||
|
||||
|
||||
def _verdict(composite: float, blockers: list[str]) -> str:
|
||||
n = len(blockers)
|
||||
if n >= 2 or composite < 50.0:
|
||||
return "NO-GO"
|
||||
if n == 1 or composite < 65.0:
|
||||
return "REDESIGN"
|
||||
if composite < 80.0:
|
||||
return "GO-WITH-CONDITIONS"
|
||||
return "GO"
|
||||
|
||||
|
||||
def evaluate(study: dict, profile_name: str, phase: int) -> dict:
|
||||
if profile_name not in PROFILES:
|
||||
raise ValueError(f"Unknown profile '{profile_name}'. Choose from {list(PROFILES)}.")
|
||||
return score_plan(study, PROFILES[profile_name], phase)
|
||||
|
||||
|
||||
def _apply_named_owners(roles: list[str], owners: dict) -> list[str]:
|
||||
"""Replace generic owner roles with 'Role (Name)' when onboarding named them."""
|
||||
role_to_key = {
|
||||
"Biostatistician": "biostatistician",
|
||||
"Medical Monitor": "medical_monitor",
|
||||
"Regulatory Owner": "regulatory_owner",
|
||||
}
|
||||
out = []
|
||||
for r in roles:
|
||||
name = owners.get(role_to_key.get(r, ""))
|
||||
out.append(f"{r} ({name})" if name else r)
|
||||
return out
|
||||
|
||||
|
||||
def _render_human(r: dict) -> str:
|
||||
lines = [f"!! {BANNER}", "", f"Study: {r['study_id']} (Phase {r['phase']})",
|
||||
f"Composite feasibility: {r['composite']}/100", f"Verdict: {r['verdict']}", ""]
|
||||
lines.append("Dimension breakdown:")
|
||||
for d, s in r["breakdown"].items():
|
||||
lines.append(f" {d:26s} {s}")
|
||||
lines.append("")
|
||||
if r["blockers"]:
|
||||
lines.append("Blockers (each can force a downgrade):")
|
||||
for b in r["blockers"]:
|
||||
lines.append(f" ! {b}")
|
||||
lines.append("")
|
||||
lines.append(f"Benchmark study cost (this profile): ${r['benchmark_cost_usd']:,.0f}")
|
||||
lines.append("Named owners who must sign the gate decision: " + ", ".join(r["named_owners"]))
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description="Score study feasibility and route a phase-gate verdict (ESTIMATE ONLY).")
|
||||
p.add_argument("--input", help="Path to JSON study plan")
|
||||
p.add_argument("--profile", default=None, choices=list(PROFILES),
|
||||
help="overrides onboarding default_profile")
|
||||
p.add_argument("--phase", type=int, default=2, choices=[1, 2, 3, 4])
|
||||
p.add_argument("--output", choices=["human", "json"], default="human")
|
||||
p.add_argument("--sample", action="store_true", help="use the embedded sample")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
conf = _cfg.load_config() if _cfg else {}
|
||||
profile = args.profile or conf.get("default_profile", "drug")
|
||||
study = SAMPLE if (args.sample or not args.input) else json.load(open(args.input))
|
||||
phase = study.get("phase", args.phase) if (args.sample or not args.input) else args.phase
|
||||
try:
|
||||
result = evaluate(study, profile, phase)
|
||||
except ValueError as e:
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 2
|
||||
result["named_owners"] = _apply_named_owners(result["named_owners"], conf.get("owners") or {})
|
||||
|
||||
if args.output == "json":
|
||||
result["_banner"] = BANNER
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(_render_human(result))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,205 @@
|
||||
#!/usr/bin/env python3
|
||||
"""sample_size_estimator.py - Closed-form sample-size / power estimates for common trial designs.
|
||||
|
||||
Stdlib-only. Deterministic. NO LLM calls. This is an ESTIMATE, not a protocol:
|
||||
every output prints a banner instructing the user to confirm with a biostatistician.
|
||||
|
||||
Supported designs (normal-approximation closed forms):
|
||||
- means two-sample comparison of means (Cohen's d effect size)
|
||||
- proportions two-sample comparison of proportions (arcsine-free normal approx)
|
||||
- survival two-arm log-rank, Schoenfeld events approximation
|
||||
|
||||
z-values come from a small built-in lookup table (no scipy dependency).
|
||||
|
||||
Usage:
|
||||
python3 sample_size_estimator.py --sample
|
||||
python3 sample_size_estimator.py --design means --effect 0.5 --alpha 0.05 --power 0.8
|
||||
python3 sample_size_estimator.py --design proportions --p1 0.30 --p2 0.45 --dropout 0.15
|
||||
python3 sample_size_estimator.py --design survival --hr 0.65 --power 0.9 --output json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
import config_loader as _cfg
|
||||
except ImportError: # pragma: no cover - skill always ships config_loader
|
||||
_cfg = None
|
||||
|
||||
BANNER = "ESTIMATE ONLY — confirm with a biostatistician before finalizing the protocol."
|
||||
|
||||
# Two-sided z for alpha, and one-sided z for power (1 - beta). Lookup avoids scipy.
|
||||
Z_ALPHA_TWO_SIDED = {0.10: 1.6449, 0.05: 1.9600, 0.025: 2.2414, 0.01: 2.5758}
|
||||
Z_POWER = {0.80: 0.8416, 0.85: 1.0364, 0.90: 1.2816, 0.95: 1.6449, 0.975: 1.9600}
|
||||
|
||||
|
||||
def _z_alpha(alpha: float) -> float:
|
||||
if alpha not in Z_ALPHA_TWO_SIDED:
|
||||
raise ValueError(f"alpha must be one of {sorted(Z_ALPHA_TWO_SIDED)} (two-sided).")
|
||||
return Z_ALPHA_TWO_SIDED[alpha]
|
||||
|
||||
|
||||
def _z_power(power: float) -> float:
|
||||
if power not in Z_POWER:
|
||||
raise ValueError(f"power must be one of {sorted(Z_POWER)}.")
|
||||
return Z_POWER[power]
|
||||
|
||||
|
||||
def _inflate(n: float, dropout: float) -> int:
|
||||
if not 0.0 <= dropout < 1.0:
|
||||
raise ValueError("dropout must be in [0, 1).")
|
||||
return math.ceil(n / (1.0 - dropout))
|
||||
|
||||
|
||||
def estimate_means(effect: float, alpha: float, power: float, allocation: float, dropout: float) -> dict:
|
||||
"""Two-sample means. effect = Cohen's d (standardized mean difference)."""
|
||||
if effect <= 0:
|
||||
raise ValueError("effect (Cohen's d) must be > 0.")
|
||||
za, zb = _z_alpha(alpha), _z_power(power)
|
||||
# Equal-n per-arm: n = 2 * ((za + zb) / d)^2 ; unequal handled via allocation ratio k.
|
||||
k = allocation
|
||||
base = ((za + zb) / effect) ** 2
|
||||
n1 = (1 + 1.0 / k) * base
|
||||
n2 = (1 + k) * base
|
||||
return {
|
||||
"design": "means",
|
||||
"effect_size_cohens_d": effect,
|
||||
"alpha_two_sided": alpha,
|
||||
"power": power,
|
||||
"allocation_ratio_k": k,
|
||||
"n_group1_raw": math.ceil(n1),
|
||||
"n_group2_raw": math.ceil(n2),
|
||||
"n_group1_with_dropout": _inflate(n1, dropout),
|
||||
"n_group2_with_dropout": _inflate(n2, dropout),
|
||||
"dropout_assumed": dropout,
|
||||
"formula": "n_i = (1 + 1/k or k) * ((z_alpha + z_beta)/d)^2",
|
||||
}
|
||||
|
||||
|
||||
def estimate_proportions(p1: float, p2: float, alpha: float, power: float, dropout: float) -> dict:
|
||||
"""Two-sample proportions, normal approximation (pooled + unpooled variance term)."""
|
||||
for p in (p1, p2):
|
||||
if not 0.0 < p < 1.0:
|
||||
raise ValueError("p1 and p2 must be in (0, 1).")
|
||||
if p1 == p2:
|
||||
raise ValueError("p1 and p2 must differ.")
|
||||
za, zb = _z_alpha(alpha), _z_power(power)
|
||||
pbar = (p1 + p2) / 2.0
|
||||
delta = abs(p1 - p2)
|
||||
num = (za * math.sqrt(2 * pbar * (1 - pbar)) + zb * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
|
||||
n_per_arm = num / (delta ** 2)
|
||||
return {
|
||||
"design": "proportions",
|
||||
"p1": p1,
|
||||
"p2": p2,
|
||||
"absolute_difference": round(delta, 4),
|
||||
"alpha_two_sided": alpha,
|
||||
"power": power,
|
||||
"n_per_arm_raw": math.ceil(n_per_arm),
|
||||
"n_per_arm_with_dropout": _inflate(n_per_arm, dropout),
|
||||
"n_total_with_dropout": 2 * _inflate(n_per_arm, dropout),
|
||||
"dropout_assumed": dropout,
|
||||
"formula": "n = [z_a*sqrt(2*pbar*qbar) + z_b*sqrt(p1q1+p2q2)]^2 / (p1-p2)^2",
|
||||
}
|
||||
|
||||
|
||||
def estimate_survival(hr: float, alpha: float, power: float, prob_event: float, dropout: float) -> dict:
|
||||
"""Two-arm log-rank, Schoenfeld events approximation + n from event probability."""
|
||||
if hr <= 0 or hr == 1.0:
|
||||
raise ValueError("hazard ratio must be > 0 and != 1.")
|
||||
if not 0.0 < prob_event <= 1.0:
|
||||
raise ValueError("prob_event (overall probability of event) must be in (0, 1].")
|
||||
za, zb = _z_alpha(alpha), _z_power(power)
|
||||
log_hr = math.log(hr)
|
||||
# Schoenfeld: total events E = 4*(za+zb)^2 / (log HR)^2 (1:1 allocation)
|
||||
events = 4.0 * ((za + zb) ** 2) / (log_hr ** 2)
|
||||
n_total = events / prob_event
|
||||
return {
|
||||
"design": "survival",
|
||||
"hazard_ratio": hr,
|
||||
"alpha_two_sided": alpha,
|
||||
"power": power,
|
||||
"required_events_raw": math.ceil(events),
|
||||
"overall_event_probability": prob_event,
|
||||
"n_total_raw": math.ceil(n_total),
|
||||
"n_total_with_dropout": _inflate(n_total, dropout),
|
||||
"dropout_assumed": dropout,
|
||||
"formula": "E = 4*(z_a+z_b)^2 / (ln HR)^2 ; n = E / P(event)",
|
||||
}
|
||||
|
||||
|
||||
def _render_human(result: dict) -> str:
|
||||
lines = [f"!! {BANNER}", "", f"Design: {result['design']}", ""]
|
||||
for k, v in result.items():
|
||||
if k == "design":
|
||||
continue
|
||||
lines.append(f" {k:28s} : {v}")
|
||||
lines += [
|
||||
"",
|
||||
"Assumptions block (state these in the protocol statistical section):",
|
||||
f" - alpha (two-sided): {result.get('alpha_two_sided')}",
|
||||
f" - power (1 - beta): {result.get('power')}",
|
||||
f" - dropout inflation: {result.get('dropout_assumed')}",
|
||||
" - The effect/difference/HR must trace to a published or anchor-based source.",
|
||||
"",
|
||||
f"Named owner required: {result.get('_biostatistician') or 'a biostatistician (run onboard.py to name one)'} "
|
||||
"must sign the final sample-size justification.",
|
||||
]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description="Closed-form clinical sample-size / power estimates (ESTIMATE ONLY).")
|
||||
p.add_argument("--design", choices=["means", "proportions", "survival"], default="means")
|
||||
p.add_argument("--alpha", type=float, default=None, help="two-sided alpha (0.10/0.05/0.025/0.01)")
|
||||
p.add_argument("--power", type=float, default=None, help="target power (0.80/0.85/0.90/0.95/0.975)")
|
||||
p.add_argument("--dropout", type=float, default=None, help="anticipated dropout fraction [0,1)")
|
||||
# means
|
||||
p.add_argument("--effect", type=float, default=0.5, help="Cohen's d (means design)")
|
||||
p.add_argument("--allocation", type=float, default=1.0, help="allocation ratio k = n2/n1 (means)")
|
||||
# proportions
|
||||
p.add_argument("--p1", type=float, default=0.30, help="control proportion")
|
||||
p.add_argument("--p2", type=float, default=0.45, help="treatment proportion")
|
||||
# survival
|
||||
p.add_argument("--hr", type=float, default=0.65, help="hazard ratio (survival)")
|
||||
p.add_argument("--prob-event", type=float, default=0.60, help="overall probability of event (survival)")
|
||||
p.add_argument("--output", choices=["human", "json"], default="human")
|
||||
p.add_argument("--sample", action="store_true", help="run the embedded sample (means design)")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
conf = _cfg.load_config() if _cfg else {}
|
||||
alpha = args.alpha if args.alpha is not None else conf.get("default_alpha", 0.05)
|
||||
power = args.power if args.power is not None else conf.get("default_power", 0.80)
|
||||
dropout = args.dropout if args.dropout is not None else conf.get("default_dropout", 0.0)
|
||||
biostat = (conf.get("owners") or {}).get("biostatistician")
|
||||
|
||||
try:
|
||||
if args.sample:
|
||||
result = estimate_means(0.5, alpha, power, 1.0, dropout if dropout else 0.15)
|
||||
elif args.design == "means":
|
||||
result = estimate_means(args.effect, alpha, power, args.allocation, dropout)
|
||||
elif args.design == "proportions":
|
||||
result = estimate_proportions(args.p1, args.p2, alpha, power, dropout)
|
||||
else:
|
||||
result = estimate_survival(args.hr, alpha, power, args.prob_event, dropout)
|
||||
except ValueError as e:
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 2
|
||||
result["_biostatistician"] = biostat
|
||||
|
||||
if args.output == "json":
|
||||
result["_banner"] = BANNER
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(_render_human(result))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,146 @@
|
||||
---
|
||||
name: market-research
|
||||
description: Use when doing upstream market-research methodology — sizing a market as TAM/SAM/SOM computed BOTH top-down and bottoms-up (never a single unsourced number), planning a survey sample size with finite-population correction and per-segment minimums, or scoring candidate market segments against Kotler's measurable/substantial/accessible/differentiable/actionable criteria. Outputs always show the method and the assumptions. For market-research analysts and product-marketing at the sizing/survey/segmentation moment. Distinct from marketing-skill (campaign analytics, attribution, demand-gen) — this is the evidence-building methodology, not live-campaign optimization.
|
||||
version: 2.9.0
|
||||
author: claude-code-skills
|
||||
license: MIT
|
||||
tags: [research-ops, market-research, tam-sam-som, market-sizing, survey, sampling, segmentation, competitive-intelligence]
|
||||
compatible_tools: [claude-code, codex-cli, cursor, antigravity, opencode, gemini-cli]
|
||||
---
|
||||
|
||||
# market-research
|
||||
|
||||
Upstream market-research methodology: market sizing, survey/sampling design, and segmentation. The discipline here is **method + assumptions**: a TAM is never a single number, a survey is never powered only in aggregate, and a segment is never a demographic slice.
|
||||
|
||||
## Purpose
|
||||
|
||||
Market-research analysts, product marketers, and strategy teams need rigorous evidence *before* anyone optimizes a campaign or sets a strategy. This skill structures three methodology decisions:
|
||||
|
||||
Three deterministic tools:
|
||||
|
||||
1. `market_sizer.py` — Computes TAM/SAM/SOM by **both** top-down and bottoms-up methods side-by-side, reports the divergence, and flags failed triangulation. Never returns a single number.
|
||||
2. `sample_size_planner.py` — Survey sample size from confidence, margin of error, and expected proportion, with the finite-population correction and **per-segment minimums** (a survey powered overall is not powered per reported segment).
|
||||
3. `segmentation_scorer.py` — Scores candidate segments against Kotler's five criteria and enforces a substantiality + accessibility gate; a slice that is too small or unreachable is dropped.
|
||||
|
||||
## When to use
|
||||
|
||||
Invoke this skill when:
|
||||
|
||||
- A board or exec asks "how big is this market?" and you need a defensible, triangulated answer.
|
||||
- You are fielding a survey and need a sample size that holds up per segment, not just overall.
|
||||
- You have a list of candidate segments and need to know which are real markets vs demographic slices.
|
||||
- You are synthesizing competitive intelligence and need a methodological backbone.
|
||||
|
||||
**Do NOT use this skill to**: measure a live campaign (attribution, ROAS, CPA → `marketing-skill/campaign-analytics`), build demand-gen / paid-media plans (`marketing-skill/marketing-demand-acquisition`), set positioning / GTM strategy (`marketing-skill/marketing-strategy-pmm`), or set pricing (`commercial/pricing-strategist`).
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Write the brief** — Fill `assets/market_research_brief_template.md` (objective, the decision this informs, sizing approach, sampling plan, assumptions register).
|
||||
2. **Size the market** — Run `market_sizer.py --input market.json --method both --profile {b2b-saas|consumer|enterprise|marketplace|hardware|services}`. Reconcile the top-down/bottoms-up delta before quoting anything.
|
||||
3. **Plan the survey** — Run `sample_size_planner.py --input survey.json`. Fund the per-segment floors, not just the overall n.
|
||||
4. **Score the segments** — Run `segmentation_scorer.py --input segments.json --profile <same>`. Drop segments failing the substantiality/accessibility gate.
|
||||
5. **Assemble the evidence pack** — Combine into a brief. Every number carries its method + assumptions + confidence.
|
||||
|
||||
## Scripts
|
||||
|
||||
| Script | Purpose | Profiles |
|
||||
|---|---|---|
|
||||
| `scripts/market_sizer.py` | TAM/SAM/SOM top-down AND bottoms-up + triangulation flag | b2b-saas, consumer, enterprise, marketplace, hardware, services |
|
||||
| `scripts/sample_size_planner.py` | Survey n + FPC + per-segment minima | n/a (parameter-driven) |
|
||||
| `scripts/segmentation_scorer.py` | Kotler 5-criteria scoring + gate | b2b-saas, consumer, enterprise, marketplace, hardware, services |
|
||||
|
||||
All three: stdlib-only, `--help`, `--sample`, `--output {human,json}`.
|
||||
|
||||
## Onboarding & customization
|
||||
|
||||
Run the onboarding questionnaire **once before you start** — it captures your defaults so every tool in this skill is pre-configured. Customization is the point: the answers actually change tool behavior.
|
||||
|
||||
```bash
|
||||
python3 scripts/onboard.py # interactive (also: --defaults, --set key=value, --reset)
|
||||
python3 scripts/onboard.py --show # see the questions + current effective config
|
||||
```
|
||||
|
||||
Answers are saved to `~/.config/research-ops/market-research.json` (global) or `./.research-ops/market-research.json` (`--scope project`) and are read automatically by `config_loader.py`. They set the default market **profile**, the default survey **confidence** and **margin of error**, and the default **sizing method**. CLI flags always override saved config; `RESEARCH_OPS_NO_CONFIG=1` ignores it.
|
||||
|
||||
**The four questions:** market profile · survey confidence · margin of error · sizing method.
|
||||
|
||||
## Optimize with autoresearch (opt-in)
|
||||
|
||||
This skill ships an **isolated, opt-in** bridge to `engineering/autoresearch-agent`. Only when you ask to "optimize" / "reconcile the sizing" / "run a loop" does an autoresearch experiment iteratively reconcile your market model so top-down and bottoms-up triangulate. `scripts/ar_evaluator.py` is the ground-truth evaluator; it prints `tam_divergence: <fraction>` (**lower** is better).
|
||||
|
||||
```bash
|
||||
/ar:setup --domain custom --name tam-triangulation \
|
||||
--target market.json \
|
||||
--eval "python3 ar_evaluator.py --target market.json" \
|
||||
--metric tam_divergence --direction lower
|
||||
/ar:loop custom/tam-triangulation
|
||||
```
|
||||
|
||||
Isolated: no hard dependency — autoresearch runs only on demand, and the loop edits `market.json`, never the evaluator.
|
||||
|
||||
## References
|
||||
|
||||
- `references/market_sizing_canon.md` — TAM/SAM/SOM frameworks (Bessemer, a16z); top-down vs bottoms-up; Fermi estimation; market-model conventions; common sizing fallacies.
|
||||
- `references/survey_methodology.md` — Cochran *Sampling Techniques*; Dillman *Tailored Design Method*; Groves *Survey Methodology*; question-wording bias (Schuman & Presser); AAPOR standards.
|
||||
- `references/segmentation_and_ci.md` — Kotler segmentation criteria; needs-based vs firmographic; Porter Five Forces; SCIP ethics; Christensen JTBD; conjoint/MaxDiff primer.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- The sizer reports both methods but cannot validate your inputs — a top-down "1% of a $40B market" is only as good as the cited source and the serviceable fraction.
|
||||
- Sample-size uses the conservative p=0.5 (maximum variance) unless you supply an expected proportion.
|
||||
- Segment scores are inputs you provide; the tool enforces the gates and the weighting, it does not gather the underlying evidence.
|
||||
- Competitive intelligence must follow the SCIP code of ethics — no misrepresentation, no protected information.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **A single TAM number with no method.** Always triangulate top-down against bottoms-up.
|
||||
- **Spurious precision.** Size to the decision's tolerance; "$3.7142B" implies a confidence you do not have.
|
||||
- **Powering only the total.** Each reported segment needs its own sample floor.
|
||||
- **Leading or double-barreled survey questions.** Pre-test wording against the bias literature.
|
||||
- **Calling a demographic slice a segment.** It must be substantial AND accessible.
|
||||
|
||||
## Distinct from
|
||||
|
||||
| Neighbor | Scope | Difference |
|
||||
|---|---|---|
|
||||
| `marketing-skill/campaign-analytics` | Attribution, ROAS, CPA, funnel of a live campaign | That **measures spend deployed**; this is **upstream methodology** |
|
||||
| `marketing-skill/marketing-demand-acquisition` | Demand-gen, paid media, channel mix | That **runs acquisition**; this **builds the evidence** |
|
||||
| `marketing-skill/marketing-strategy-pmm` | Positioning, GTM, category | That **sets strategy**; this **sizes and segments the market** |
|
||||
| `commercial/pricing-strategist` | Pricing model + WTP + packaging | That **sets price**; this **sizes the market** |
|
||||
| `product-research` (sibling) | User/product discovery methods | That studies **users**; this studies **the market** |
|
||||
|
||||
## Quick examples
|
||||
|
||||
```bash
|
||||
python3 scripts/market_sizer.py --sample
|
||||
python3 scripts/sample_size_planner.py --population 62000 --confidence 0.95 --moe 0.05
|
||||
python3 scripts/segmentation_scorer.py --sample --output json
|
||||
```
|
||||
|
||||
The sample market triangulates a ~$1.47B top-down SAM against the bottoms-up figure and flags the divergence; the segmentation sample drops the "solopreneurs who might want analytics" slice for failing the substantiality and accessibility gates.
|
||||
|
||||
## Forcing-question library (Matt Pocock grill discipline)
|
||||
|
||||
Walked one at a time by `/cs:grill-research-ops` or the orchestrator. Recommended answer + canon citation per question. Never bundled.
|
||||
|
||||
1. **"Is your TAM top-down or bottoms-up — and have you computed it both ways to triangulate?"**
|
||||
Recommended: both; reconcile the delta before quoting a number.
|
||||
Canon: Bessemer / a16z market-sizing; Fermi estimation.
|
||||
|
||||
2. **"What decision will this market size actually drive — and at what precision does it matter?"**
|
||||
Recommended: size to the decision's tolerance, not to a spurious-precision number.
|
||||
Canon: market-model conventions (Gartner/Forrester); decision-driven analysis.
|
||||
|
||||
3. **"What's your target margin of error and confidence — and does your sample clear it per segment, not just overall?"**
|
||||
Recommended: power each reported segment, not only the total.
|
||||
Canon: Cochran *Sampling Techniques*; AAPOR standards.
|
||||
|
||||
4. **"Are your survey questions free of leading and double-barreled wording?"**
|
||||
Recommended: pre-test the wording; cite the bias source.
|
||||
Canon: Schuman & Presser; Dillman *Tailored Design Method*.
|
||||
|
||||
5. **"Do your segments pass measurable / substantial / accessible / actionable — or are they just demographic slices?"**
|
||||
Recommended: drop segments that fail substantiality or accessibility.
|
||||
Canon: Kotler segmentation criteria.
|
||||
|
||||
Walk depth-first. Lock 1-2 before opening 3-5. After all are answered, invoke `market_sizer.py` → `sample_size_planner.py` → `segmentation_scorer.py`.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Market Research Brief — Template
|
||||
|
||||
> Fill this before running the tools. Every number in the final brief must carry its method,
|
||||
> assumptions, and confidence. Size to the decision's tolerance, not to false precision.
|
||||
|
||||
## 1. Objective
|
||||
- Research question:
|
||||
- **The decision this informs** (and who makes it):
|
||||
- Precision required (order-of-magnitude / ±20% / ±5%):
|
||||
|
||||
## 2. Market sizing
|
||||
- Approach: [top-down | bottoms-up | both — recommended both]
|
||||
- Top-down inputs: total market value + source citation; serviceable fraction; reachable share.
|
||||
- Bottoms-up inputs: total potential customers + source; annual price; serviceable fraction; realistic adoption.
|
||||
- Triangulation result (from `market_sizer.py`) + divergence:
|
||||
|
||||
## 3. Survey plan (if primary data)
|
||||
- Population (N) + sampling frame:
|
||||
- Confidence level / overall margin of error / expected proportion:
|
||||
- Segments to report + per-segment margin of error:
|
||||
- Recommended n (overall + per-segment floors, from `sample_size_planner.py`):
|
||||
- Mode (online panel / phone / mixed) + coverage risk:
|
||||
|
||||
## 4. Segmentation
|
||||
- Candidate segments + scores across measurable / substantial / accessible / differentiable / actionable:
|
||||
- Verdicts (from `segmentation_scorer.py`): TARGET / WATCH / DROP
|
||||
|
||||
## 5. Competitive intelligence
|
||||
- Sources (public, ethically obtained — SCIP code):
|
||||
- Five Forces summary:
|
||||
|
||||
## 6. Assumptions register
|
||||
- (List every assumption behind the sizing, sampling, and segmentation. Each must trace to a source or be flagged as an unverified planning assumption.)
|
||||
|
||||
## 7. Confidence statement
|
||||
- Overall confidence in the headline numbers (high / moderate / low) and why:
|
||||
@@ -0,0 +1,36 @@
|
||||
# Market Sizing Canon
|
||||
|
||||
Reference for TAM/SAM/SOM. Pairs with `market_sizer.py`.
|
||||
|
||||
## The three numbers
|
||||
|
||||
- **TAM (Total Addressable Market)** — total revenue if you captured 100% of the market for your category.
|
||||
- **SAM (Serviceable Addressable Market)** — the portion of TAM you can serve given your geography, segment focus, and product scope.
|
||||
- **SOM (Serviceable Obtainable Market)** — the realistic, capacity-constrained share of SAM you can win in the planning window.
|
||||
|
||||
## Two methods — always do both
|
||||
|
||||
- **Top-down**: start from a published total market value (analyst report, government statistic) and apply serviceable and reachable fractions. Fast, but only as good as the source and inherits its biases. The classic failure is "1% of a huge number" — a TAM that sounds enormous and means nothing.
|
||||
- **Bottoms-up**: start from the number of potential customers × the price they would pay, then apply serviceable and adoption fractions. Slower but grounded in units you can defend. The discipline is that bottoms-up forces you to name customer counts and price points.
|
||||
|
||||
When the two methods diverge by more than a tolerance, **triangulation has failed** — you do not yet have a defensible number. The tool flags this rather than averaging the two (averaging hides the disagreement).
|
||||
|
||||
## Fermi discipline
|
||||
|
||||
Good sizing is Fermi estimation: decompose the unknown into knowable factors, estimate each with a stated assumption, and carry the uncertainty through. A market size is a chain of assumptions; surfacing the chain is the deliverable, not the point estimate.
|
||||
|
||||
## Common fallacies
|
||||
|
||||
- **Double counting** — summing overlapping segments or counting the same revenue at multiple layers of the value chain.
|
||||
- **Percent-of-a-big-number** — anchoring on "if we just get 1%" without a bottoms-up cross-check.
|
||||
- **Confusing TAM growth with your growth** — a growing TAM does not entitle you to a fixed share.
|
||||
- **Spurious precision** — quoting a TAM to four significant figures when the inputs are order-of-magnitude estimates.
|
||||
|
||||
## Sources
|
||||
|
||||
1. Bessemer Venture Partners, *State of the Cloud* and market-sizing memos (TAM/SAM/SOM discipline).
|
||||
2. Andreessen Horowitz (a16z), *The truth about market sizing* and bottoms-up TAM essays.
|
||||
3. Gartner / Forrester / IDC market-model methodology notes (forecast construction conventions).
|
||||
4. Weinstein, L., & Adam, J., *Guesstimation* (Princeton, 2008) — Fermi estimation.
|
||||
5. Blank, S., *The Four Steps to the Epiphany* — market-type and sizing in customer development.
|
||||
6. Damodaran, A., *Narrative and Numbers* (Columbia, 2017) — disciplining market-size narratives with numbers.
|
||||
@@ -0,0 +1,43 @@
|
||||
# Segmentation and Competitive Intelligence
|
||||
|
||||
Reference for segmentation scoring and competitive-intelligence synthesis. Pairs with `segmentation_scorer.py`.
|
||||
|
||||
## What makes a segment useful (Kotler)
|
||||
|
||||
A market segment is only useful if it meets five criteria. The scorer weights and gates them:
|
||||
|
||||
1. **Measurable** — you can size and identify it.
|
||||
2. **Substantial** — it is large and profitable enough to be worth serving. (Gate: a tiny slice is not a market.)
|
||||
3. **Accessible** — you can reach it through channels you can afford. (Gate: an unreachable segment is academic.)
|
||||
4. **Differentiable** — it responds differently to your offer than other segments do; otherwise it is not a distinct segment.
|
||||
5. **Actionable** — you can design and execute a program for it.
|
||||
|
||||
Substantiality and accessibility are **gates** because they are the two that most often fail silently: teams fall in love with a precisely-described segment that is too small or has no viable channel.
|
||||
|
||||
## Bases of segmentation
|
||||
|
||||
- **Firmographic** (B2B): industry, size, geography — easy to measure, weak at predicting behavior.
|
||||
- **Demographic** (B2C): age, income, role — same trade-off.
|
||||
- **Needs-based / behavioral**: grouping by the job customers are trying to get done. Stronger predictor of response, harder to measure. Christensen's **Jobs-to-be-Done** reframes segmentation around the progress a customer is trying to make, not who they are.
|
||||
|
||||
The strongest segmentations pair a needs-based core with a firmographic/demographic proxy you can actually target.
|
||||
|
||||
## Competitive intelligence
|
||||
|
||||
CI synthesis is structured, ethical analysis of the competitive landscape:
|
||||
|
||||
- **Porter's Five Forces** frames structural attractiveness (rivalry, new entrants, substitutes, supplier power, buyer power).
|
||||
- **SCIP (Strategic and Competitive Intelligence Professionals)** publishes a code of ethics: no misrepresentation of identity, no acquisition of protected/confidential information, full compliance with law. CI is built from public and ethically-obtained sources.
|
||||
|
||||
## Advanced preference measurement
|
||||
|
||||
When you need to quantify trade-offs, **conjoint analysis** and **MaxDiff** (best-worst scaling) estimate the relative importance of attributes and the willingness to trade one for another — far more reliable than directly asking "how important is X?"
|
||||
|
||||
## Sources
|
||||
|
||||
1. Kotler, P., & Keller, K., *Marketing Management*, 15th ed. — segmentation criteria.
|
||||
2. Christensen, Hall, Dillon & Duncan, *Competing Against Luck* (2016) — Jobs-to-be-Done.
|
||||
3. Porter, M., *Competitive Strategy* (1980) — Five Forces.
|
||||
4. SCIP, *Code of Ethics for CI Professionals*.
|
||||
5. Orme, B., *Getting Started with Conjoint Analysis*, 4th ed. (Sawtooth, Research Publishers).
|
||||
6. Smith, W., *Product Differentiation and Market Segmentation as Alternative Marketing Strategies* — J Marketing 1956 (the founding segmentation paper).
|
||||
@@ -0,0 +1,39 @@
|
||||
# Survey Methodology
|
||||
|
||||
Reference for survey design and sampling. Pairs with `sample_size_planner.py`.
|
||||
|
||||
## Sample size from first principles
|
||||
|
||||
For estimating a proportion, the required sample is n₀ = z²·p·(1−p)/e², where z is the critical value for the confidence level, p the expected proportion, and e the margin of error. The maximum-variance choice p = 0.5 is the conservative default. When the sample is a meaningful fraction of a finite population N, apply the **finite-population correction**: n = n₀ / (1 + (n₀−1)/N). The planner does both.
|
||||
|
||||
The most common analyst error: powering the **overall** sample but then reporting **per-segment** results that the sample cannot support. If you will report three segments at ±8%, each segment needs ~150 respondents — so the survey must be sized to the segment floors, not the aggregate. The planner computes per-segment minimums explicitly.
|
||||
|
||||
## The total survey error framework
|
||||
|
||||
Sample size only addresses **sampling error**. Groves' total-survey-error framework names the others, often larger:
|
||||
|
||||
- **Coverage error** — the sampling frame omits part of the population (e.g., an email panel misses non-users).
|
||||
- **Non-response error** — those who respond differ systematically from those who don't.
|
||||
- **Measurement error** — the instrument itself biases answers (question wording, order, scale).
|
||||
|
||||
A tight margin of error on a biased frame is precision without accuracy.
|
||||
|
||||
## Question design
|
||||
|
||||
- **Avoid leading questions** that imply a preferred answer.
|
||||
- **Avoid double-barreled questions** that ask two things at once ("Is the product fast and reliable?").
|
||||
- **Watch scale and order effects** — response options and question sequence shift answers.
|
||||
- **Pre-test** every instrument with a small cognitive-interview pass before fielding.
|
||||
|
||||
## Standards
|
||||
|
||||
AAPOR (American Association for Public Opinion Research) publishes disclosure standards and the standard definitions for response-rate calculation. Reputable market research follows them so results are comparable and auditable.
|
||||
|
||||
## Sources
|
||||
|
||||
1. Cochran, W.G., *Sampling Techniques*, 3rd ed. (Wiley, 1977).
|
||||
2. Dillman, Smyth & Christian, *Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method*, 4th ed. (2014).
|
||||
3. Groves et al., *Survey Methodology*, 2nd ed. (Wiley, 2009) — total survey error.
|
||||
4. Schuman, H., & Presser, S., *Questions and Answers in Attitude Surveys* (1981) — wording/order effects.
|
||||
5. AAPOR, *Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys*.
|
||||
6. Tourangeau, Rips & Rasinski, *The Psychology of Survey Response* (Cambridge, 2000).
|
||||
@@ -0,0 +1,77 @@
|
||||
#!/usr/bin/env python3
|
||||
"""ar_evaluator.py - Autoresearch evaluator for the market-research skill (OPT-IN).
|
||||
|
||||
Stdlib-only. The ISOLATED bridge to engineering/autoresearch-agent. It does NOT call
|
||||
autoresearch; it is the ground-truth evaluator an autoresearch loop runs after editing
|
||||
the target market model. It reads a market-model JSON, runs market_sizer in "both" mode,
|
||||
and prints ONE metric line:
|
||||
|
||||
tam_divergence: <fraction> (LOWER is better — top-down and bottoms-up should agree)
|
||||
|
||||
Optimize a market model so the two sizing methods triangulate (reconcile assumptions),
|
||||
while the agent edits the target. The user opts in explicitly:
|
||||
/ar:setup --domain custom --name tam-triangulation \\
|
||||
--target market.json --eval "python3 ar_evaluator.py --target market.json" \\
|
||||
--metric tam_divergence --direction lower
|
||||
|
||||
Direct use:
|
||||
python3 ar_evaluator.py --sample
|
||||
python3 ar_evaluator.py --target market.json --profile enterprise
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import config_loader as cfg # noqa: E402
|
||||
import market_sizer as ms # noqa: E402
|
||||
|
||||
METRIC = "tam_divergence"
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
c = cfg.load_config()
|
||||
p = argparse.ArgumentParser(description="Autoresearch evaluator: TAM triangulation divergence.")
|
||||
p.add_argument("--target", help="path to market-model JSON (or env AR_TARGET)")
|
||||
p.add_argument("--profile", default=None, help="overrides onboarding default_profile")
|
||||
p.add_argument("--sample", action="store_true")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
profile = args.profile or c.get("default_profile", "b2b-saas")
|
||||
if args.sample:
|
||||
data = ms.SAMPLE
|
||||
else:
|
||||
target = args.target or os.environ.get("AR_TARGET")
|
||||
if not target:
|
||||
print("error: provide --target <market.json> or set AR_TARGET", file=sys.stderr)
|
||||
return 2
|
||||
try:
|
||||
with open(target) as f:
|
||||
data = json.load(f)
|
||||
except (OSError, json.JSONDecodeError) as e:
|
||||
print(f"{METRIC}: N/A")
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
try:
|
||||
result = ms.size_market(data, "both", profile)
|
||||
except ValueError as e:
|
||||
print(f"{METRIC}: N/A")
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
div = result.get("tam_divergence")
|
||||
if div is None:
|
||||
print(f"{METRIC}: N/A")
|
||||
print("error: need both top_down and bottoms_up blocks to triangulate", file=sys.stderr)
|
||||
return 1
|
||||
print(f"{METRIC}: {div}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,114 @@
|
||||
#!/usr/bin/env python3
|
||||
"""config_loader.py - Customization loader for the market-research skill.
|
||||
|
||||
Stdlib-only. Importable from the skill's other scripts. Precedence (highest wins):
|
||||
1. Project config: <cwd>/.research-ops/market-research.json
|
||||
2. Global config: ~/.config/research-ops/market-research.json
|
||||
3. Built-in DEFAULTS
|
||||
|
||||
Onboarding answers (written by onboard.py) live in these files; every tool in this
|
||||
skill reads them so the user's customization applies automatically.
|
||||
Set RESEARCH_OPS_NO_CONFIG=1 to ignore saved config.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
SKILL = "market-research"
|
||||
|
||||
GLOBAL_CONFIG_DIR = Path.home() / ".config" / "research-ops"
|
||||
GLOBAL_CONFIG_PATH = GLOBAL_CONFIG_DIR / f"{SKILL}.json"
|
||||
PROJECT_CONFIG_DIRNAME = ".research-ops"
|
||||
|
||||
DEFAULTS: dict[str, Any] = {
|
||||
"version": 1,
|
||||
"skill": SKILL,
|
||||
"default_profile": "b2b-saas",
|
||||
"default_confidence": 0.95,
|
||||
"default_moe": 0.05,
|
||||
"sizing_method": "both",
|
||||
"setup_completed_at": None,
|
||||
}
|
||||
|
||||
|
||||
def project_config_path(cwd: Path | None = None) -> Path:
|
||||
cwd = cwd or Path.cwd()
|
||||
return cwd / PROJECT_CONFIG_DIRNAME / f"{SKILL}.json"
|
||||
|
||||
|
||||
def _read_json(path: Path) -> dict[str, Any] | None:
|
||||
try:
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
return data if isinstance(data, dict) else None
|
||||
except (FileNotFoundError, json.JSONDecodeError, OSError):
|
||||
return None
|
||||
|
||||
|
||||
def _deep_merge(base: dict[str, Any], override: dict[str, Any]) -> dict[str, Any]:
|
||||
out = dict(base)
|
||||
for k, v in override.items():
|
||||
if isinstance(v, dict) and isinstance(out.get(k), dict):
|
||||
out[k] = _deep_merge(out[k], v)
|
||||
else:
|
||||
out[k] = v
|
||||
return out
|
||||
|
||||
|
||||
def load_config(cwd: Path | None = None) -> dict[str, Any]:
|
||||
config = dict(DEFAULTS)
|
||||
if os.environ.get("RESEARCH_OPS_NO_CONFIG") == "1":
|
||||
return config
|
||||
global_cfg = _read_json(GLOBAL_CONFIG_PATH)
|
||||
if global_cfg:
|
||||
config = _deep_merge(config, global_cfg)
|
||||
project_cfg = _read_json(project_config_path(cwd))
|
||||
if project_cfg:
|
||||
config = _deep_merge(config, project_cfg)
|
||||
return config
|
||||
|
||||
|
||||
def setup_completed() -> bool:
|
||||
cfg = _read_json(GLOBAL_CONFIG_PATH) or _read_json(project_config_path())
|
||||
return bool(cfg and cfg.get("setup_completed_at"))
|
||||
|
||||
|
||||
def write_config(config: dict[str, Any], scope: str = "global", cwd: Path | None = None) -> Path:
|
||||
path = project_config_path(cwd) if scope == "project" else GLOBAL_CONFIG_PATH
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("w", encoding="utf-8") as f:
|
||||
json.dump(config, f, indent=2, sort_keys=True)
|
||||
return path
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description=f"Inspect {SKILL} customization config.")
|
||||
p.add_argument("--show", action="store_true", help="Print the effective config")
|
||||
p.add_argument("--status", action="store_true", help="Print setup status + paths")
|
||||
p.add_argument("--sample", action="store_true", help="Print the built-in defaults")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
if args.sample:
|
||||
print(json.dumps(DEFAULTS, indent=2, sort_keys=True))
|
||||
elif args.status:
|
||||
print(json.dumps({
|
||||
"skill": SKILL,
|
||||
"global_config_path": str(GLOBAL_CONFIG_PATH),
|
||||
"global_config_exists": GLOBAL_CONFIG_PATH.exists(),
|
||||
"project_config_path": str(project_config_path()),
|
||||
"project_config_exists": project_config_path().exists(),
|
||||
"setup_completed": setup_completed(),
|
||||
}, indent=2))
|
||||
else:
|
||||
print(json.dumps(load_config(), indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,162 @@
|
||||
#!/usr/bin/env python3
|
||||
"""market_sizer.py - Compute TAM / SAM / SOM by BOTH top-down and bottoms-up methods.
|
||||
|
||||
Stdlib-only. Deterministic. NO LLM calls. NEVER returns a single number: it computes both
|
||||
methods side-by-side, reports the delta, and prints a mandatory method + assumptions block.
|
||||
|
||||
Top-down: TAM = total_market_value ; SAM = TAM * serviceable_fraction ; SOM = SAM * reachable_share
|
||||
Bottoms-up: TAM = total_potential_customers * annual_price
|
||||
SAM = TAM * serviceable_fraction
|
||||
SOM = SAM * realistic_adoption (capacity-constrained)
|
||||
|
||||
If the two TAMs diverge by more than the tolerance, the tool flags it: triangulation failed.
|
||||
|
||||
Usage:
|
||||
python3 market_sizer.py --sample
|
||||
python3 market_sizer.py --input market.json --method both
|
||||
python3 market_sizer.py --input market.json --profile b2b-saas --output json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
import config_loader as _cfg
|
||||
except ImportError: # pragma: no cover
|
||||
_cfg = None
|
||||
|
||||
# Profiles tune the divergence tolerance and a sanity note.
|
||||
PROFILES = {
|
||||
"b2b-saas": {"tolerance": 0.30},
|
||||
"consumer": {"tolerance": 0.40},
|
||||
"enterprise": {"tolerance": 0.25},
|
||||
"marketplace": {"tolerance": 0.40},
|
||||
"hardware": {"tolerance": 0.30},
|
||||
"services": {"tolerance": 0.35},
|
||||
}
|
||||
|
||||
SAMPLE = {
|
||||
"market_name": "Mid-market HR analytics SaaS (US)",
|
||||
"top_down": {
|
||||
"total_market_value": 4200000000,
|
||||
"serviceable_fraction": 0.35,
|
||||
"reachable_share": 0.04,
|
||||
},
|
||||
"bottoms_up": {
|
||||
"total_potential_customers": 62000,
|
||||
"annual_price": 18000,
|
||||
"serviceable_fraction": 0.35,
|
||||
"realistic_adoption": 0.03,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def top_down(td: dict) -> dict:
|
||||
tam = float(td.get("total_market_value", 0.0))
|
||||
sam = tam * float(td.get("serviceable_fraction", 0.0))
|
||||
som = sam * float(td.get("reachable_share", 0.0))
|
||||
return {"method": "top-down", "TAM": round(tam, 0), "SAM": round(sam, 0), "SOM": round(som, 0)}
|
||||
|
||||
|
||||
def bottoms_up(bu: dict) -> dict:
|
||||
customers = float(bu.get("total_potential_customers", 0.0))
|
||||
price = float(bu.get("annual_price", 0.0))
|
||||
tam = customers * price
|
||||
sam = tam * float(bu.get("serviceable_fraction", 0.0))
|
||||
som = sam * float(bu.get("realistic_adoption", 0.0))
|
||||
return {"method": "bottoms-up", "TAM": round(tam, 0), "SAM": round(sam, 0),
|
||||
"SOM": round(som, 0), "implied_customers_at_SOM": round((sam / price) * float(bu.get("realistic_adoption", 0.0))) if price else None}
|
||||
|
||||
|
||||
def size_market(data: dict, method: str, profile: str) -> dict:
|
||||
if profile not in PROFILES:
|
||||
raise ValueError(f"Unknown profile '{profile}'. Choose from {list(PROFILES)}.")
|
||||
tol = PROFILES[profile]["tolerance"]
|
||||
out = {"market_name": data.get("market_name", "UNSPECIFIED"), "profile": profile}
|
||||
|
||||
td = top_down(data.get("top_down", {})) if method in ("top-down", "both") else None
|
||||
bu = bottoms_up(data.get("bottoms_up", {})) if method in ("bottoms-up", "both") else None
|
||||
if td:
|
||||
out["top_down"] = td
|
||||
if bu:
|
||||
out["bottoms_up"] = bu
|
||||
|
||||
flags = []
|
||||
if td and bu and td["TAM"] > 0:
|
||||
delta = abs(td["TAM"] - bu["TAM"]) / td["TAM"]
|
||||
out["tam_divergence"] = round(delta, 3)
|
||||
if delta > tol:
|
||||
flags.append(f"TRIANGULATION FAILED: top-down and bottoms-up TAM differ by {delta:.0%} "
|
||||
f"(> {tol:.0%} tolerance). Reconcile before quoting a number.")
|
||||
else:
|
||||
flags.append(f"Triangulation OK: TAMs within {delta:.0%} (tolerance {tol:.0%}).")
|
||||
out["flags"] = flags
|
||||
out["method_and_assumptions"] = [
|
||||
"NEVER quote a single TAM number without stating the method and the assumptions below.",
|
||||
"Top-down TAM = total market value (cite the source: analyst report, gov stat).",
|
||||
"Bottoms-up TAM = total potential customers x annual price (cite both counts).",
|
||||
"SAM = TAM x serviceable fraction (geography/segment you can actually serve).",
|
||||
"SOM = SAM x realistic, capacity-constrained share you can win in the planning window.",
|
||||
]
|
||||
return out
|
||||
|
||||
|
||||
def _fmt(n):
|
||||
return f"${n:,.0f}" if isinstance(n, (int, float)) else str(n)
|
||||
|
||||
|
||||
def _render_human(r: dict) -> str:
|
||||
lines = [f"Market Sizing: {r['market_name']} (profile: {r['profile']})", ""]
|
||||
for key in ("top_down", "bottoms_up"):
|
||||
if key in r:
|
||||
m = r[key]
|
||||
lines.append(f" [{m['method']}] TAM {_fmt(m['TAM'])} | SAM {_fmt(m['SAM'])} | SOM {_fmt(m['SOM'])}")
|
||||
if m.get("implied_customers_at_SOM") is not None:
|
||||
lines.append(f" implied customers at SOM: {m['implied_customers_at_SOM']:,}")
|
||||
if "tam_divergence" in r:
|
||||
lines.append(f" TAM divergence (top-down vs bottoms-up): {r['tam_divergence']:.1%}")
|
||||
lines.append("")
|
||||
for f in r["flags"]:
|
||||
lines.append(f" ! {f}")
|
||||
lines.append("")
|
||||
lines.append("Method & assumptions (must travel with the number):")
|
||||
for a in r["method_and_assumptions"]:
|
||||
lines.append(f" - {a}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description="Compute TAM/SAM/SOM by top-down AND bottoms-up (never a single number).")
|
||||
p.add_argument("--input", help="Path to JSON with top_down{} and bottoms_up{}")
|
||||
p.add_argument("--method", choices=["top-down", "bottoms-up", "both"], default=None,
|
||||
help="overrides onboarding sizing_method")
|
||||
p.add_argument("--profile", default=None, choices=list(PROFILES),
|
||||
help="overrides onboarding default_profile")
|
||||
p.add_argument("--output", choices=["human", "json"], default="human")
|
||||
p.add_argument("--sample", action="store_true", help="use the embedded sample")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
conf = _cfg.load_config() if _cfg else {}
|
||||
method = args.method or conf.get("sizing_method", "both")
|
||||
profile = args.profile or conf.get("default_profile", "b2b-saas")
|
||||
data = SAMPLE if (args.sample or not args.input) else json.load(open(args.input))
|
||||
try:
|
||||
result = size_market(data, method, profile)
|
||||
except ValueError as e:
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
if args.output == "json":
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(_render_human(result))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,117 @@
|
||||
#!/usr/bin/env python3
|
||||
"""onboard.py - Onboarding questionnaire for the market-research skill.
|
||||
|
||||
Stdlib-only. Asks the user a short set of questions BEFORE they size a market or field
|
||||
a survey, then writes the answers to a customization config read by every tool in this
|
||||
skill via config_loader.py. The answers become defaults for profile, survey confidence,
|
||||
margin of error, and sizing method.
|
||||
|
||||
Modes: --show | --defaults | --set key=value (repeatable) | --reset | --scope {global,project}
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as _dt
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import config_loader as cfg # noqa: E402
|
||||
|
||||
NUMERIC_KEYS = {"default_confidence", "default_moe"}
|
||||
|
||||
QUESTIONS = [
|
||||
("default_profile",
|
||||
"1. What market are you researching?",
|
||||
["b2b-saas", "consumer", "enterprise", "marketplace", "hardware", "services"], str),
|
||||
("default_confidence",
|
||||
"2. Default survey confidence level?",
|
||||
["0.80", "0.85", "0.90", "0.95", "0.99"], float),
|
||||
("default_moe",
|
||||
"3. Default survey margin of error (fraction, e.g. 0.05)?",
|
||||
None, float),
|
||||
("sizing_method",
|
||||
"4. Default market-sizing method?",
|
||||
["top-down", "bottoms-up", "both"], str),
|
||||
]
|
||||
|
||||
|
||||
def _print_questions() -> None:
|
||||
print(f"Onboarding questions — {cfg.SKILL}:\n")
|
||||
for _k, prompt, choices, _c in QUESTIONS:
|
||||
line = f" {prompt}"
|
||||
if choices:
|
||||
line += f" [{' / '.join(choices)}]"
|
||||
print(line)
|
||||
|
||||
|
||||
def run_interactive(config: dict) -> dict:
|
||||
print(f"Onboarding — {cfg.SKILL}. Press Enter to keep the current/default value.\n")
|
||||
for key, prompt, choices, caster in QUESTIONS:
|
||||
suffix = f" [{'/'.join(choices)}]" if choices else ""
|
||||
cur = f" (current: {config.get(key)})" if config.get(key) is not None else ""
|
||||
raw = input(f"{prompt}{suffix}{cur}: ").strip()
|
||||
if not raw:
|
||||
continue
|
||||
try:
|
||||
config[key] = caster(raw)
|
||||
except ValueError:
|
||||
print(f" ! invalid value for {key}, keeping current")
|
||||
return config
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description=f"Onboarding for the {cfg.SKILL} skill.")
|
||||
p.add_argument("--show", action="store_true")
|
||||
p.add_argument("--defaults", action="store_true", help="write built-in defaults, no prompt")
|
||||
p.add_argument("--set", action="append", default=[], metavar="key=value")
|
||||
p.add_argument("--reset", action="store_true")
|
||||
p.add_argument("--scope", choices=["global", "project"], default="global")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
if args.show:
|
||||
_print_questions()
|
||||
print("\nCurrent effective config:")
|
||||
print(json.dumps(cfg.load_config(), indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
if args.reset:
|
||||
path = cfg.project_config_path() if args.scope == "project" else cfg.GLOBAL_CONFIG_PATH
|
||||
if path.exists():
|
||||
path.unlink(); print(f"removed {path}")
|
||||
else:
|
||||
print(f"no config at {path}")
|
||||
return 0
|
||||
|
||||
config = cfg.load_config()
|
||||
|
||||
if args.set:
|
||||
for item in args.set:
|
||||
if "=" not in item:
|
||||
print(f"error: --set expects key=value, got '{item}'", file=sys.stderr)
|
||||
return 2
|
||||
k, v = item.split("=", 1)
|
||||
if k in NUMERIC_KEYS:
|
||||
try:
|
||||
v = float(v)
|
||||
except ValueError:
|
||||
pass
|
||||
config[k] = v
|
||||
elif not args.defaults:
|
||||
if sys.stdin.isatty():
|
||||
config = run_interactive(config)
|
||||
else:
|
||||
print("non-interactive shell: use --defaults or --set key=value. Showing questions:\n")
|
||||
_print_questions()
|
||||
return 0
|
||||
|
||||
config["setup_completed_at"] = _dt.datetime.now(_dt.timezone.utc).isoformat()
|
||||
path = cfg.write_config(config, scope=args.scope)
|
||||
print(f"saved {cfg.SKILL} customization -> {path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,183 @@
|
||||
#!/usr/bin/env python3
|
||||
"""sample_size_planner.py - Survey sample-size with finite-population correction + per-segment minima.
|
||||
|
||||
Stdlib-only. Deterministic. NO LLM calls.
|
||||
|
||||
Computes the classic proportion-estimate sample size:
|
||||
n0 = z^2 * p * (1-p) / e^2
|
||||
then applies the finite-population correction (FPC) when a population N is given:
|
||||
n = n0 / (1 + (n0 - 1)/N)
|
||||
Also computes per-segment minimums and a proportional quota allocation, because a survey
|
||||
powered overall is NOT powered per reported segment.
|
||||
|
||||
Usage:
|
||||
python3 sample_size_planner.py --sample
|
||||
python3 sample_size_planner.py --population 62000 --confidence 0.95 --moe 0.05 --proportion 0.5
|
||||
python3 sample_size_planner.py --input survey.json --output json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
import config_loader as _cfg
|
||||
except ImportError: # pragma: no cover
|
||||
_cfg = None
|
||||
|
||||
Z = {0.80: 1.2816, 0.85: 1.4395, 0.90: 1.6449, 0.95: 1.9600, 0.99: 2.5758}
|
||||
|
||||
SAMPLE = {
|
||||
"population": 62000,
|
||||
"confidence": 0.95,
|
||||
"margin_of_error": 0.05,
|
||||
"expected_proportion": 0.5,
|
||||
"segments": [
|
||||
{"name": "1-50 employees", "population_share": 0.55},
|
||||
{"name": "51-250 employees", "population_share": 0.30},
|
||||
{"name": "251-1000 employees", "population_share": 0.15},
|
||||
],
|
||||
"segment_moe": 0.08,
|
||||
}
|
||||
|
||||
|
||||
def _z(conf: float) -> float:
|
||||
if conf not in Z:
|
||||
raise ValueError(f"confidence must be one of {sorted(Z)}.")
|
||||
return Z[conf]
|
||||
|
||||
|
||||
def base_n(conf: float, moe: float, p: float, population: float | None) -> dict:
|
||||
if not 0.0 < p < 1.0:
|
||||
raise ValueError("expected_proportion must be in (0,1).")
|
||||
if not 0.0 < moe < 1.0:
|
||||
raise ValueError("margin_of_error must be in (0,1).")
|
||||
z = _z(conf)
|
||||
n0 = (z ** 2) * p * (1 - p) / (moe ** 2)
|
||||
if population and population > 0:
|
||||
n = n0 / (1 + (n0 - 1) / population)
|
||||
fpc_applied = True
|
||||
else:
|
||||
n = n0
|
||||
fpc_applied = False
|
||||
return {
|
||||
"confidence": conf,
|
||||
"margin_of_error": moe,
|
||||
"expected_proportion": p,
|
||||
"population": population,
|
||||
"n_unadjusted": math.ceil(n0),
|
||||
"n_with_fpc": math.ceil(n),
|
||||
"fpc_applied": fpc_applied,
|
||||
"z": z,
|
||||
}
|
||||
|
||||
|
||||
def segment_plan(data: dict, overall: dict) -> dict:
|
||||
seg_moe = float(data.get("segment_moe", data.get("margin_of_error", 0.05)))
|
||||
conf = float(data.get("confidence", 0.95))
|
||||
p = float(data.get("expected_proportion", 0.5))
|
||||
z = _z(conf)
|
||||
per_seg_min = math.ceil((z ** 2) * p * (1 - p) / (seg_moe ** 2))
|
||||
|
||||
segs = data.get("segments", [])
|
||||
total_quota = max(overall["n_with_fpc"], per_seg_min * len(segs)) if segs else overall["n_with_fpc"]
|
||||
out_segs = []
|
||||
for s in segs:
|
||||
share = float(s.get("population_share", 0.0))
|
||||
proportional = math.ceil(total_quota * share)
|
||||
quota = max(proportional, per_seg_min)
|
||||
out_segs.append({
|
||||
"name": s.get("name", "UNNAMED"),
|
||||
"population_share": share,
|
||||
"proportional_quota": proportional,
|
||||
"minimum_for_segment_moe": per_seg_min,
|
||||
"recommended_quota": quota,
|
||||
})
|
||||
return {
|
||||
"segment_margin_of_error": seg_moe,
|
||||
"minimum_per_segment": per_seg_min,
|
||||
"recommended_total_with_segment_floors": sum(s["recommended_quota"] for s in out_segs) if out_segs else total_quota,
|
||||
"segments": out_segs,
|
||||
}
|
||||
|
||||
|
||||
def plan(data: dict) -> dict:
|
||||
overall = base_n(
|
||||
float(data.get("confidence", 0.95)),
|
||||
float(data.get("margin_of_error", 0.05)),
|
||||
float(data.get("expected_proportion", 0.5)),
|
||||
data.get("population"),
|
||||
)
|
||||
result = {"overall": overall}
|
||||
if data.get("segments"):
|
||||
result["segmentation"] = segment_plan(data, overall)
|
||||
result["notes"] = [
|
||||
"A survey powered overall is NOT powered per reported segment — fund the segment floors.",
|
||||
"expected_proportion=0.5 is the conservative (maximum-variance) default.",
|
||||
"FPC matters when the sample is a large fraction of the population (small N).",
|
||||
]
|
||||
return result
|
||||
|
||||
|
||||
def _render_human(r: dict) -> str:
|
||||
o = r["overall"]
|
||||
lines = ["Survey Sample-Size Plan", "",
|
||||
f" Confidence: {o['confidence']:.0%} MoE: {o['margin_of_error']:.0%} p: {o['expected_proportion']}",
|
||||
f" n (unadjusted): {o['n_unadjusted']}",
|
||||
f" n (with FPC): {o['n_with_fpc']} (population {o['population']}, fpc_applied={o['fpc_applied']})"]
|
||||
if "segmentation" in r:
|
||||
s = r["segmentation"]
|
||||
lines += ["", f" Per-segment MoE: {s['segment_margin_of_error']:.0%} => minimum {s['minimum_per_segment']} per segment",
|
||||
f" Recommended total with segment floors: {s['recommended_total_with_segment_floors']}", ""]
|
||||
for seg in s["segments"]:
|
||||
lines.append(f" {seg['name']:24s} share {seg['population_share']:.0%} "
|
||||
f"proportional {seg['proportional_quota']} recommended {seg['recommended_quota']}")
|
||||
lines += ["", "Notes:"]
|
||||
for n in r["notes"]:
|
||||
lines.append(f" - {n}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description="Survey sample size with FPC + per-segment minima.")
|
||||
p.add_argument("--input", help="Path to JSON survey spec")
|
||||
p.add_argument("--population", type=float, default=None)
|
||||
p.add_argument("--confidence", type=float, default=None, help="overrides onboarding default_confidence")
|
||||
p.add_argument("--moe", type=float, default=None, help="margin of error (overrides onboarding default_moe)")
|
||||
p.add_argument("--proportion", type=float, default=0.5, help="expected proportion")
|
||||
p.add_argument("--output", choices=["human", "json"], default="human")
|
||||
p.add_argument("--sample", action="store_true", help="use the embedded sample")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
conf = _cfg.load_config() if _cfg else {}
|
||||
confidence = args.confidence if args.confidence is not None else conf.get("default_confidence", 0.95)
|
||||
moe = args.moe if args.moe is not None else conf.get("default_moe", 0.05)
|
||||
|
||||
if args.sample:
|
||||
data = SAMPLE
|
||||
elif args.input:
|
||||
data = json.load(open(args.input))
|
||||
else:
|
||||
data = {"population": args.population, "confidence": confidence,
|
||||
"margin_of_error": moe, "expected_proportion": args.proportion}
|
||||
|
||||
try:
|
||||
result = plan(data)
|
||||
except ValueError as e:
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
if args.output == "json":
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(_render_human(result))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,139 @@
|
||||
#!/usr/bin/env python3
|
||||
"""segmentation_scorer.py - Score candidate market segments against Kotler's actionability criteria.
|
||||
|
||||
Stdlib-only. Deterministic. NO LLM calls.
|
||||
|
||||
Each candidate segment is scored 0-100 across the five Kotler criteria for a useful segment:
|
||||
1. measurable can you size and identify it?
|
||||
2. substantial is it large/profitable enough to serve?
|
||||
3. accessible can you reach it through channels?
|
||||
4. differentiable does it respond differently from other segments?
|
||||
5. actionable can you design and execute a program for it?
|
||||
|
||||
Segments failing the substantiality or accessibility gates are flagged: a demographic slice
|
||||
that is unreachable or too small is not a market segment.
|
||||
|
||||
Usage:
|
||||
python3 segmentation_scorer.py --sample
|
||||
python3 segmentation_scorer.py --input segments.json --profile enterprise
|
||||
python3 segmentation_scorer.py --input segments.json --output json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
import config_loader as _cfg
|
||||
except ImportError: # pragma: no cover
|
||||
_cfg = None
|
||||
|
||||
CRITERIA = ["measurable", "substantial", "accessible", "differentiable", "actionable"]
|
||||
WEIGHTS = {"measurable": 0.15, "substantial": 0.25, "accessible": 0.25,
|
||||
"differentiable": 0.20, "actionable": 0.15}
|
||||
GATE_FLOOR = 40.0 # substantiality + accessibility gate
|
||||
|
||||
# Profiles nudge the substantiality expectation (enterprise tolerates smaller, higher-value segments).
|
||||
PROFILES = {
|
||||
"b2b-saas": 1.0,
|
||||
"consumer": 1.0,
|
||||
"enterprise": 0.85,
|
||||
"marketplace": 1.0,
|
||||
"hardware": 1.0,
|
||||
"services": 0.9,
|
||||
}
|
||||
|
||||
SAMPLE = {
|
||||
"segments": [
|
||||
{"name": "Mid-market HR teams (51-250 emp)",
|
||||
"scores": {"measurable": 85, "substantial": 80, "accessible": 75, "differentiable": 70, "actionable": 80}},
|
||||
{"name": "Solopreneurs who 'might' want analytics",
|
||||
"scores": {"measurable": 40, "substantial": 30, "accessible": 35, "differentiable": 30, "actionable": 40}},
|
||||
{"name": "Enterprise CHROs (1000+ emp)",
|
||||
"scores": {"measurable": 90, "substantial": 95, "accessible": 50, "differentiable": 85, "actionable": 70}},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def score_segment(seg: dict, sub_mult: float) -> dict:
|
||||
raw = seg.get("scores", {})
|
||||
breakdown = {}
|
||||
composite = 0.0
|
||||
for c in CRITERIA:
|
||||
s = float(raw.get(c, 0.0))
|
||||
if c == "substantial":
|
||||
s = min(100.0, s / sub_mult) # enterprise: smaller segments still count (divide by <1 raises)
|
||||
breakdown[c] = round(s, 1)
|
||||
composite += s * WEIGHTS[c]
|
||||
flags = []
|
||||
if breakdown["substantial"] < GATE_FLOOR:
|
||||
flags.append("FAILS SUBSTANTIALITY GATE: too small/unprofitable to be a target segment.")
|
||||
if breakdown["accessible"] < GATE_FLOOR:
|
||||
flags.append("FAILS ACCESSIBILITY GATE: no viable channel to reach it.")
|
||||
verdict = "DROP" if flags else ("TARGET" if composite >= 65 else "WATCH")
|
||||
return {
|
||||
"name": seg.get("name", "UNNAMED"),
|
||||
"composite": round(composite, 1),
|
||||
"breakdown": breakdown,
|
||||
"flags": flags,
|
||||
"verdict": verdict,
|
||||
}
|
||||
|
||||
|
||||
def evaluate(data: dict, profile: str) -> dict:
|
||||
if profile not in PROFILES:
|
||||
raise ValueError(f"Unknown profile '{profile}'. Choose from {list(PROFILES)}.")
|
||||
mult = PROFILES[profile]
|
||||
scored = sorted((score_segment(s, mult) for s in data.get("segments", [])),
|
||||
key=lambda x: x["composite"], reverse=True)
|
||||
return {
|
||||
"profile": profile,
|
||||
"segments": scored,
|
||||
"note": "A demographic or firmographic slice is not a segment unless it is substantial AND accessible.",
|
||||
}
|
||||
|
||||
|
||||
def _render_human(r: dict) -> str:
|
||||
lines = [f"Segmentation Scoring (profile: {r['profile']})", ""]
|
||||
for s in r["segments"]:
|
||||
lines.append(f"[{s['verdict']}] {s['name']} — composite {s['composite']}/100")
|
||||
for c, v in s["breakdown"].items():
|
||||
lines.append(f" {c:16s} {v}")
|
||||
for f in s["flags"]:
|
||||
lines.append(f" ! {f}")
|
||||
lines.append("")
|
||||
lines.append(f"note: {r['note']}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description="Score market segments against Kotler's 5 criteria.")
|
||||
p.add_argument("--input", help="Path to JSON with segments[]")
|
||||
p.add_argument("--profile", default=None, choices=list(PROFILES),
|
||||
help="overrides onboarding default_profile")
|
||||
p.add_argument("--output", choices=["human", "json"], default="human")
|
||||
p.add_argument("--sample", action="store_true", help="use the embedded sample")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
conf = _cfg.load_config() if _cfg else {}
|
||||
profile = args.profile or conf.get("default_profile", "b2b-saas")
|
||||
data = SAMPLE if (args.sample or not args.input) else json.load(open(args.input))
|
||||
try:
|
||||
result = evaluate(data, profile)
|
||||
except ValueError as e:
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
if args.output == "json":
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(_render_human(result))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,145 @@
|
||||
---
|
||||
name: product-research
|
||||
description: Use when planning and synthesizing product/user research as a method-and-repository discipline — selecting the right method for the goal (generative interviews vs usability test vs concept test vs validation), computing method-based saturation/sample size with an explicit confidence level, or synthesizing coded observations into insights while flagging single-source anecdotes. Never fabricates user insight; an insight requires recurrence across independent participants. Distinct from product-team/ux-researcher-designer (persona/journey artifacts), product-discovery (discovery-sprint planning), and experiment-designer (live A/B) — this is the research-ops method + insight-repository layer.
|
||||
version: 2.9.0
|
||||
author: claude-code-skills
|
||||
license: MIT
|
||||
tags: [research-ops, product-research, ux-research, jtbd, usability, saturation, insight-synthesis, research-repository]
|
||||
compatible_tools: [claude-code, codex-cli, cursor, antigravity, opencode, gemini-cli]
|
||||
---
|
||||
|
||||
# product-research
|
||||
|
||||
Product / user research as an operational discipline: choosing the right method, sizing it honestly, and synthesizing findings into governed insights. The core rule: **method must match the goal**, and **an insight requires recurrence across independent participants** — a single quote is an anecdote.
|
||||
|
||||
## Purpose
|
||||
|
||||
Product researchers, ResearchOps teams, and PMs running discovery need method rigor and an insight repository they can trust. This skill structures three decisions:
|
||||
|
||||
Three deterministic tools:
|
||||
|
||||
1. `study_designer.py` — Maps (research goal × product stage) to an appropriate method and emits a method-matched plan skeleton (objective, participant criteria, guide structure, success criteria). Redirects live A/B to `product-team/experiment-designer`.
|
||||
2. `saturation_planner.py` — Method-based sample guidance with an explicit **confidence label**: Nielsen problem-discovery (5/segment), Guest et al. thematic saturation (~12), and evaluative coverage. Never claims a prevalence rate from a small-n usability test.
|
||||
3. `insight_synthesizer.py` — Clusters coded observations by tag, counts distinct participants, ranks by cross-participant recurrence, and flags any candidate below the source threshold as an **ANECDOTE**, never promoting it to an insight.
|
||||
|
||||
## When to use
|
||||
|
||||
Invoke this skill when:
|
||||
|
||||
- You are planning a study and need the method to match the goal (generative vs evaluative vs validation).
|
||||
- You need a defensible sample size / saturation rationale with a stated confidence.
|
||||
- You have raw coded observations and need to synthesize insights without over-claiming.
|
||||
- You are setting up or auditing a research repository and need the insight-vs-observation discipline.
|
||||
|
||||
**Do NOT use this skill to**: generate personas / journey maps (use `product-team/ux-researcher-designer`), plan a discovery sprint or validate an opportunity (use `product-team/product-discovery`), design or analyze a live product A/B experiment (use `product-team/experiment-designer`), or do market sizing / surveys (use the `market-research` sibling).
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Frame the study** — Fill `assets/research_plan_template.md` (research questions, method rationale, participant criteria, analysis plan, repository tagging scheme).
|
||||
2. **Pick the method** — Run `study_designer.py --goal {discovery|evaluative|validation} --stage {concept|prototype|beta|live} --profile {b2b-saas|consumer-app|enterprise|marketplace|hardware|platform}`. Honor the redirect if it routes to experiment-designer.
|
||||
3. **Size it** — Run `saturation_planner.py --method {usability|thematic|evaluative-coverage} --segments N`. Record the confidence label and limits.
|
||||
4. **Synthesize** — After fielding, code observations and run `insight_synthesizer.py --input observations.json --min-sources 3`. Treat ANECDOTE-flagged clusters as signals to probe, not findings to ship.
|
||||
5. **File in the repository** — Tag insights to the atomic schema at synthesis time, with their evidence and confidence.
|
||||
|
||||
## Scripts
|
||||
|
||||
| Script | Purpose | Profiles |
|
||||
|---|---|---|
|
||||
| `scripts/study_designer.py` | (goal × stage) → method + plan skeleton | b2b-saas, consumer-app, enterprise, marketplace, hardware, platform |
|
||||
| `scripts/saturation_planner.py` | Method-based sample guidance + confidence | n/a (method-driven) |
|
||||
| `scripts/insight_synthesizer.py` | Cluster observations, flag anecdotes | n/a (evidence-driven) |
|
||||
|
||||
All three: stdlib-only, `--help`, `--sample`, `--output {human,json}`.
|
||||
|
||||
## Onboarding & customization
|
||||
|
||||
Run the onboarding questionnaire **once before you start** — it captures your defaults so every tool in this skill is pre-configured. Customization is the point: the answers actually change tool behavior (e.g. the insight source-threshold).
|
||||
|
||||
```bash
|
||||
python3 scripts/onboard.py # interactive (also: --defaults, --set key=value, --reset)
|
||||
python3 scripts/onboard.py --show # see the questions + current effective config
|
||||
```
|
||||
|
||||
Answers are saved to `~/.config/research-ops/product-research.json` (global) or `./.research-ops/product-research.json` (`--scope project`) and are read automatically by `config_loader.py`. They set the default product **profile**, the **insight source-threshold** (how many independent participants make a finding an insight, not an anecdote), the default **saturation method**, and the **high-stakes** flag. CLI flags always override saved config; `RESEARCH_OPS_NO_CONFIG=1` ignores it.
|
||||
|
||||
**The four questions:** product profile · insight source-threshold · saturation method · high-stakes flag.
|
||||
|
||||
## Optimize with autoresearch (opt-in)
|
||||
|
||||
This skill ships an **isolated, opt-in** bridge to `engineering/autoresearch-agent`. Only when you ask to "optimize the synthesis" / "run a loop" does an autoresearch experiment iteratively refine the coding/clustering of a fixed evidence set so more cross-participant patterns surface. `scripts/ar_evaluator.py` is the ground-truth evaluator; it prints `validated_insights: <int>` (higher is better). It optimizes the **coding**, never fabricates evidence.
|
||||
|
||||
```bash
|
||||
/ar:setup --domain custom --name insight-synthesis \
|
||||
--target observations.json \
|
||||
--eval "python3 ar_evaluator.py --target observations.json" \
|
||||
--metric validated_insights --direction higher
|
||||
/ar:loop custom/insight-synthesis
|
||||
```
|
||||
|
||||
Isolated: no hard dependency — autoresearch runs only on demand, and the loop edits `observations.json`, never the evaluator.
|
||||
|
||||
## References
|
||||
|
||||
- `references/research_methods_canon.md` — Portigal *Interviewing Users*; Christensen/Ulwick JTBD; Rohrer's UX-research methods landscape (NN/g); Sauro & Lewis *Quantifying the User Experience*; Goodman/Kuniavsky.
|
||||
- `references/sampling_and_saturation.md` — Nielsen "test with 5 users"; Guest, Bunce & Johnson saturation; Faulkner on more-than-5; Sauro usability sample size; Braun & Clarke thematic analysis.
|
||||
- `references/repository_and_synthesis.md` — ResearchOps / atomic research (Tomer Sharon "Polaris"); insight-vs-observation discipline; repository governance; affinity mapping; democratization guardrails.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- Method selection assumes you can name the goal honestly; if the goal is fuzzy, grill it first (the goal drives everything).
|
||||
- Saturation guidance is method-based, not a power calculation — usability tests find problems, not prevalence rates.
|
||||
- The synthesizer counts evidence you provide; coding quality is upstream of it. Garbage tags → garbage clusters.
|
||||
- The insight threshold (`--min-sources`) defaults to 3; raise it for high-stakes or heterogeneous populations.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Mismatching method to goal.** A usability test cannot discover unmet needs; an interview cannot measure task success.
|
||||
- **Reporting usability problems as percentages.** Small-n tests surface problems, not population rates.
|
||||
- **Promoting an anecdote to an insight.** One participant is a signal to probe, not a finding.
|
||||
- **Framing interview questions as feature reactions.** Probe the job-to-be-done and recent real behavior, not hypothetical opinions.
|
||||
- **Synthesizing without a repository scheme.** Tag at synthesis time, or insights rot unfindable.
|
||||
|
||||
## Distinct from
|
||||
|
||||
| Neighbor | Scope | Difference |
|
||||
|---|---|---|
|
||||
| `product-team/ux-researcher-designer` | Personas, journey maps, usability frameworks tied to design output | That produces **artifacts**; this is **method + repository discipline** |
|
||||
| `product-team/product-discovery` | Opportunity validation, discovery-sprint planning | That plans **discovery sprints**; this designs and synthesizes the **research** |
|
||||
| `product-team/experiment-designer` | Live product A/B hypothesis + sample size | That runs **live experiments**; this runs **qualitative/evaluative research** |
|
||||
| `market-research` (sibling) | Market sizing, surveys, segmentation | That studies **the market**; this studies **users** |
|
||||
|
||||
## Quick examples
|
||||
|
||||
```bash
|
||||
python3 scripts/study_designer.py --sample
|
||||
python3 scripts/saturation_planner.py --method thematic --segments 3
|
||||
python3 scripts/insight_synthesizer.py --sample --min-sources 3
|
||||
```
|
||||
|
||||
The synthesizer sample correctly promotes "import-confusion" (3 independent participants) to INSIGHT and flags "wants-slack" (1 participant) as an ANECDOTE.
|
||||
|
||||
## Forcing-question library (Matt Pocock grill discipline)
|
||||
|
||||
Walked one at a time by `/cs:grill-research-ops` or the orchestrator. Recommended answer + canon citation per question. Never bundled.
|
||||
|
||||
1. **"Is this study generative (discover problems) or evaluative (test a solution)?"**
|
||||
Recommended: name it first — the method follows from the goal.
|
||||
Canon: Rohrer, *When to Use Which User-Experience Research Methods* (NN/g).
|
||||
|
||||
2. **"What's your sample size and saturation rationale — and at what confidence?"**
|
||||
Recommended: method-based n (5/segment usability; ~12 for thematic saturation), state the confidence.
|
||||
Canon: Nielsen; Guest, Bunce & Johnson (2006); Faulkner (2003).
|
||||
|
||||
3. **"How many independent participants support each insight — or is it a single-source anecdote?"**
|
||||
Recommended: require recurrence across ≥3 sources before calling it an insight; flag singletons.
|
||||
Canon: atomic research / ResearchOps; Braun & Clarke thematic analysis.
|
||||
|
||||
4. **"Are your interview / usability tasks framed as outcomes (jobs) or as feature reactions?"**
|
||||
Recommended: frame around the job-to-be-done and recent real behavior, not hypothetical opinion.
|
||||
Canon: Christensen/Ulwick Jobs-to-be-Done; Portigal *Interviewing Users*.
|
||||
|
||||
5. **"Where does this land in the repository, and how is it tagged for reuse?"**
|
||||
Recommended: tag to the atomic schema at synthesis time, not later.
|
||||
Canon: Tomer Sharon, *Polaris* / ResearchOps repository practice.
|
||||
|
||||
Walk depth-first. Lock 1-2 before opening 3-5. After all are answered, invoke `study_designer.py` → `saturation_planner.py` → (after fielding) `insight_synthesizer.py`.
|
||||
@@ -0,0 +1,50 @@
|
||||
# Product Research Plan — Template
|
||||
|
||||
> Fill this before running the tools. Method must match the goal. An insight requires
|
||||
> recurrence across independent participants — a single quote is an anecdote.
|
||||
|
||||
## 1. Study identification
|
||||
- Study name:
|
||||
- Product / feature:
|
||||
- Stage: [concept | prototype | beta | live]
|
||||
- Profile: [b2b-saas | consumer-app | enterprise | marketplace | hardware | platform]
|
||||
|
||||
## 2. Goal & questions
|
||||
- Goal: [discovery (generative) | evaluative | validation]
|
||||
- Research questions (3-5, answerable, not leading):
|
||||
- The product decision this informs:
|
||||
|
||||
## 3. Method (from `study_designer.py`)
|
||||
- Recommended method:
|
||||
- Why it matches the goal:
|
||||
- (If live A/B → route to product-team/experiment-designer.)
|
||||
|
||||
## 4. Participants
|
||||
- Target segment(s) + screener (screen for the job, not a job title):
|
||||
- Per-segment recruiting if reporting per segment? [yes/no]
|
||||
- Exclusions (internal, biased, repeat):
|
||||
|
||||
## 5. Sample & saturation (from `saturation_planner.py`)
|
||||
- Method: [usability | thematic | evaluative-coverage]
|
||||
- n per segment + total:
|
||||
- Confidence label + limits:
|
||||
|
||||
## 6. Study guide skeleton
|
||||
1.
|
||||
2.
|
||||
3.
|
||||
4.
|
||||
5.
|
||||
|
||||
## 7. Analysis & synthesis
|
||||
- Coding / tagging scheme (atomic taxonomy):
|
||||
- Insight threshold (min distinct participants): ___ (default 3)
|
||||
- Synthesis tool: `insight_synthesizer.py`
|
||||
|
||||
## 8. Repository
|
||||
- Where insights are filed + tagging taxonomy:
|
||||
- Evidence linked to each insight? [yes — required]
|
||||
- Confidence field per insight? [yes — required]
|
||||
|
||||
## 9. Confidence statement
|
||||
- What this study can and cannot support:
|
||||
@@ -0,0 +1,28 @@
|
||||
# Research Repository and Synthesis
|
||||
|
||||
Reference for turning observations into governed insights. Pairs with `insight_synthesizer.py`.
|
||||
|
||||
## Observation vs insight
|
||||
|
||||
The foundational discipline of ResearchOps is the distinction between an **observation** (a single piece of evidence — one participant did or said one thing) and an **insight** (a pattern that recurs across independent sources and carries an implication). Promoting an observation to an insight because it was vivid or confirmed a prior is the cardinal sin of synthesis. The synthesizer enforces a source threshold: a candidate supported by fewer than the threshold of distinct participants is labeled an ANECDOTE and is never promoted.
|
||||
|
||||
## Atomic research
|
||||
|
||||
Tomer Sharon's **atomic research** model (and the "Polaris" repository concept) decomposes research into reusable units: *Experiments → Facts (observations) → Insights → Recommendations*. Facts are tagged and stored so that insights can be traced back to evidence and reused across studies. The payoff is a repository where a claim can always be drilled down to the observations that support it — and where the same evidence can support future questions.
|
||||
|
||||
## Affinity mapping
|
||||
|
||||
The classic synthesis technique is affinity mapping: cluster observations into emergent themes bottom-up, then name the themes. The `insight_synthesizer.py` tool is a deterministic, tag-based proxy for this — it clusters by the codes you assign and ranks by cross-participant recurrence. The human still does the interpretive naming; the tool enforces the counting discipline.
|
||||
|
||||
## Repository governance and democratization
|
||||
|
||||
As organizations democratize research (PMs and designers running their own studies), the repository becomes the guardrail. Governance practices: a consistent tagging taxonomy, evidence linked to every insight, a confidence field, and a review step before an insight is marked "validated." Without governance, democratized research produces a pile of unsearchable anecdotes; with it, the repository compounds in value.
|
||||
|
||||
## Sources
|
||||
|
||||
1. Sharon, T., *Validating Product Ideas Through Lean User Research* (Rosenfeld, 2016) and the atomic-research / Polaris model.
|
||||
2. ResearchOps Community, *Research Repositories* and *Democratization* working-group reports.
|
||||
3. Braun, V., & Clarke, V., *Thematic Analysis: A Practical Guide* (Sage, 2022).
|
||||
4. Beyer, H., & Holtzblatt, K., *Contextual Design* (1998) — affinity diagramming.
|
||||
5. Dovetail / EnjoyHQ practitioner guides on insight repositories and tagging taxonomies.
|
||||
6. Kaplan, K., *Taxonomy 101* and *Research Repositories* — Nielsen Norman Group.
|
||||
@@ -0,0 +1,34 @@
|
||||
# Product Research Methods Canon
|
||||
|
||||
Reference for method selection. Pairs with `study_designer.py`.
|
||||
|
||||
## The two-axis map
|
||||
|
||||
UX/product research methods sort along two axes (Rohrer, NN/g): **attitudinal vs behavioral** (what people say vs what they do) and **qualitative vs quantitative** (why/how vs how-many). The single most important pre-method decision is the **goal**:
|
||||
|
||||
- **Generative (discovery)** — you don't yet know the problem. Methods: semi-structured interviews, contextual inquiry, diary studies. Output: themes, unmet needs, jobs-to-be-done.
|
||||
- **Evaluative** — you have a solution and want to know if it works. Methods: moderated/unmoderated usability tests, concept tests. Output: task-success, severity-rated problems.
|
||||
- **Validation** — you want to confirm demand/desirability before building. Methods: surveys, preference tests, fake-door tests, and (when live) A/B experiments.
|
||||
|
||||
Picking an evaluative method for a generative goal — "let's usability-test our way to product strategy" — is the most common and most expensive error.
|
||||
|
||||
## Interviewing discipline
|
||||
|
||||
Steve Portigal's *Interviewing Users* is the operative craft reference: ask about **recent, specific, real behavior** ("tell me about the last time you…"), not hypotheticals or opinions ("would you use…"). People are unreliable narrators of their future selves but good storytellers of their past.
|
||||
|
||||
## Jobs-to-be-Done
|
||||
|
||||
Christensen's and Ulwick's JTBD reframes research around the **progress a person is trying to make** in a circumstance, not their demographics or feature preferences. Outcome-Driven Innovation (Ulwick) operationalizes this into measurable desired outcomes — a bridge between qualitative discovery and quantitative validation.
|
||||
|
||||
## Mixed methods
|
||||
|
||||
Strong research triangulates: qualitative discovery surfaces hypotheses; quantitative validation sizes them. Sauro & Lewis (*Quantifying the User Experience*) provides the statistical backbone for turning usability observations into defensible metrics (task time, completion, SUS) without over-claiming from small samples.
|
||||
|
||||
## Sources
|
||||
|
||||
1. Portigal, S., *Interviewing Users*, 2nd ed. (Rosenfeld, 2023).
|
||||
2. Christensen, Hall, Dillon & Duncan, *Competing Against Luck* (2016) — Jobs-to-be-Done.
|
||||
3. Ulwick, A., *Jobs to Be Done: Theory to Practice* (2016) — Outcome-Driven Innovation.
|
||||
4. Rohrer, C., *When to Use Which User-Experience Research Methods* — Nielsen Norman Group.
|
||||
5. Sauro, J., & Lewis, J., *Quantifying the User Experience*, 2nd ed. (Morgan Kaufmann, 2016).
|
||||
6. Goodman, Kuniavsky & Moed, *Observing the User Experience*, 2nd ed. (2012).
|
||||
@@ -0,0 +1,29 @@
|
||||
# Sampling and Saturation
|
||||
|
||||
Reference for how many participants. Pairs with `saturation_planner.py`.
|
||||
|
||||
## Usability: the "5 users" result
|
||||
|
||||
Nielsen and Landauer's model says the proportion of usability problems found with n users is 1 − (1 − p)ⁿ, where p is the average probability that a single user surfaces a given problem (~0.31 in their data). At n = 5, that is ~85% of problems — hence "test with 5 users." Two crucial caveats the planner enforces:
|
||||
|
||||
1. **Per segment.** The 5-user result holds *within a homogeneous user group*. If you have distinct segments that behave differently, you need ~5 per segment.
|
||||
2. **Problems, not rates.** A small-n usability test finds *whether* a problem exists; it cannot estimate the *prevalence* of that problem in the population. Never report "60% of users struggled" from a 5-person test.
|
||||
|
||||
Faulkner (2003) showed real variance: while the average across many 5-person samples is ~85%, individual 5-person runs ranged from ~55% to 100%. When stakes or heterogeneity are high, run more.
|
||||
|
||||
## Qualitative: thematic saturation
|
||||
|
||||
For interview-based thematic research, Guest, Bunce & Johnson (2006) found that **saturation** — the point where new interviews stop yielding new themes — typically occurs by ~12 interviews in a homogeneous group, with the basic elements present by ~6. Saturation is **observed, not guaranteed**: track the new-theme rate and stop when it flattens, rather than committing to a fixed n blindly. Heterogeneous populations need more, and per-group saturation applies just as in usability.
|
||||
|
||||
## Reporting confidence honestly
|
||||
|
||||
The planner attaches a confidence label (LOW / MODERATE / MODERATE-HIGH) and explicit limits to every plan, because the failure mode in product research is not too-small samples per se — it is **over-claiming** from whatever sample you ran. State the method, the n, and what the method can and cannot support.
|
||||
|
||||
## Sources
|
||||
|
||||
1. Nielsen, J., & Landauer, T., *A mathematical model of the finding of usability problems* — INTERCHI 1993.
|
||||
2. Nielsen, J., *Why You Only Need to Test with 5 Users* — NN/g (2000).
|
||||
3. Faulkner, L., *Beyond the five-user assumption* — Behavior Research Methods 2003;35:379-383.
|
||||
4. Guest, G., Bunce, A., & Johnson, L., *How many interviews are enough?* — Field Methods 2006;18:59-82.
|
||||
5. Braun, V., & Clarke, V., *Using thematic analysis in psychology* — Qual Res Psychol 2006;3:77-101.
|
||||
6. Sauro, J., & Lewis, J., *Quantifying the User Experience*, 2nd ed. (2016) — confidence intervals for small samples.
|
||||
@@ -0,0 +1,68 @@
|
||||
#!/usr/bin/env python3
|
||||
"""ar_evaluator.py - Autoresearch evaluator for the product-research skill (OPT-IN).
|
||||
|
||||
Stdlib-only. The ISOLATED bridge to engineering/autoresearch-agent. It does NOT call
|
||||
autoresearch; it is the ground-truth evaluator an autoresearch loop runs after editing
|
||||
the target coded-observations file. It reads an observations JSON, runs insight_synthesizer
|
||||
at the configured source threshold, and prints ONE metric line:
|
||||
|
||||
validated_insights: <int> (higher is better — clusters that clear the source threshold)
|
||||
|
||||
This optimizes the CODING/synthesis of a fixed evidence set (merging/splitting tags so
|
||||
cross-participant patterns surface) — not the evidence itself. The user opts in explicitly:
|
||||
/ar:setup --domain custom --name insight-synthesis \\
|
||||
--target observations.json --eval "python3 ar_evaluator.py --target observations.json" \\
|
||||
--metric validated_insights --direction higher
|
||||
|
||||
Direct use:
|
||||
python3 ar_evaluator.py --sample
|
||||
python3 ar_evaluator.py --target observations.json --min-sources 3
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import config_loader as cfg # noqa: E402
|
||||
import insight_synthesizer as isyn # noqa: E402
|
||||
|
||||
METRIC = "validated_insights"
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
c = cfg.load_config()
|
||||
p = argparse.ArgumentParser(description="Autoresearch evaluator: count of validated insights.")
|
||||
p.add_argument("--target", help="path to observations JSON (or env AR_TARGET)")
|
||||
p.add_argument("--min-sources", type=int, default=None, help="overrides onboarding insight_min_sources")
|
||||
p.add_argument("--sample", action="store_true")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
min_sources = args.min_sources if args.min_sources is not None else int(c.get("insight_min_sources", 3))
|
||||
|
||||
if args.sample:
|
||||
data = isyn.SAMPLE
|
||||
else:
|
||||
target = args.target or os.environ.get("AR_TARGET")
|
||||
if not target:
|
||||
print("error: provide --target <observations.json> or set AR_TARGET", file=sys.stderr)
|
||||
return 2
|
||||
try:
|
||||
with open(target) as f:
|
||||
data = json.load(f)
|
||||
except (OSError, json.JSONDecodeError) as e:
|
||||
print(f"{METRIC}: N/A")
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
result = isyn.synthesize(data, min_sources)
|
||||
count = sum(1 for c2 in result["candidates"] if c2["classification"] == "INSIGHT")
|
||||
print(f"{METRIC}: {count}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,114 @@
|
||||
#!/usr/bin/env python3
|
||||
"""config_loader.py - Customization loader for the product-research skill.
|
||||
|
||||
Stdlib-only. Importable from the skill's other scripts. Precedence (highest wins):
|
||||
1. Project config: <cwd>/.research-ops/product-research.json
|
||||
2. Global config: ~/.config/research-ops/product-research.json
|
||||
3. Built-in DEFAULTS
|
||||
|
||||
Onboarding answers (written by onboard.py) live in these files; every tool in this
|
||||
skill reads them so the user's customization applies automatically.
|
||||
Set RESEARCH_OPS_NO_CONFIG=1 to ignore saved config.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
SKILL = "product-research"
|
||||
|
||||
GLOBAL_CONFIG_DIR = Path.home() / ".config" / "research-ops"
|
||||
GLOBAL_CONFIG_PATH = GLOBAL_CONFIG_DIR / f"{SKILL}.json"
|
||||
PROJECT_CONFIG_DIRNAME = ".research-ops"
|
||||
|
||||
DEFAULTS: dict[str, Any] = {
|
||||
"version": 1,
|
||||
"skill": SKILL,
|
||||
"default_profile": "b2b-saas",
|
||||
"insight_min_sources": 3,
|
||||
"default_method": "usability",
|
||||
"stakes_high": False,
|
||||
"setup_completed_at": None,
|
||||
}
|
||||
|
||||
|
||||
def project_config_path(cwd: Path | None = None) -> Path:
|
||||
cwd = cwd or Path.cwd()
|
||||
return cwd / PROJECT_CONFIG_DIRNAME / f"{SKILL}.json"
|
||||
|
||||
|
||||
def _read_json(path: Path) -> dict[str, Any] | None:
|
||||
try:
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
return data if isinstance(data, dict) else None
|
||||
except (FileNotFoundError, json.JSONDecodeError, OSError):
|
||||
return None
|
||||
|
||||
|
||||
def _deep_merge(base: dict[str, Any], override: dict[str, Any]) -> dict[str, Any]:
|
||||
out = dict(base)
|
||||
for k, v in override.items():
|
||||
if isinstance(v, dict) and isinstance(out.get(k), dict):
|
||||
out[k] = _deep_merge(out[k], v)
|
||||
else:
|
||||
out[k] = v
|
||||
return out
|
||||
|
||||
|
||||
def load_config(cwd: Path | None = None) -> dict[str, Any]:
|
||||
config = dict(DEFAULTS)
|
||||
if os.environ.get("RESEARCH_OPS_NO_CONFIG") == "1":
|
||||
return config
|
||||
global_cfg = _read_json(GLOBAL_CONFIG_PATH)
|
||||
if global_cfg:
|
||||
config = _deep_merge(config, global_cfg)
|
||||
project_cfg = _read_json(project_config_path(cwd))
|
||||
if project_cfg:
|
||||
config = _deep_merge(config, project_cfg)
|
||||
return config
|
||||
|
||||
|
||||
def setup_completed() -> bool:
|
||||
cfg = _read_json(GLOBAL_CONFIG_PATH) or _read_json(project_config_path())
|
||||
return bool(cfg and cfg.get("setup_completed_at"))
|
||||
|
||||
|
||||
def write_config(config: dict[str, Any], scope: str = "global", cwd: Path | None = None) -> Path:
|
||||
path = project_config_path(cwd) if scope == "project" else GLOBAL_CONFIG_PATH
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("w", encoding="utf-8") as f:
|
||||
json.dump(config, f, indent=2, sort_keys=True)
|
||||
return path
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description=f"Inspect {SKILL} customization config.")
|
||||
p.add_argument("--show", action="store_true", help="Print the effective config")
|
||||
p.add_argument("--status", action="store_true", help="Print setup status + paths")
|
||||
p.add_argument("--sample", action="store_true", help="Print the built-in defaults")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
if args.sample:
|
||||
print(json.dumps(DEFAULTS, indent=2, sort_keys=True))
|
||||
elif args.status:
|
||||
print(json.dumps({
|
||||
"skill": SKILL,
|
||||
"global_config_path": str(GLOBAL_CONFIG_PATH),
|
||||
"global_config_exists": GLOBAL_CONFIG_PATH.exists(),
|
||||
"project_config_path": str(project_config_path()),
|
||||
"project_config_exists": project_config_path().exists(),
|
||||
"setup_completed": setup_completed(),
|
||||
}, indent=2))
|
||||
else:
|
||||
print(json.dumps(load_config(), indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,113 @@
|
||||
#!/usr/bin/env python3
|
||||
"""insight_synthesizer.py - Cluster coded observations into candidate insights; flag anecdotes.
|
||||
|
||||
Stdlib-only. Deterministic. NO LLM calls. NEVER fabricates an insight: it counts evidence,
|
||||
clusters by tag, ranks by cross-participant recurrence, and flags any candidate supported by
|
||||
fewer than --min-sources independent participants as an ANECDOTE, not an insight.
|
||||
|
||||
Input: a list of observations, each with {participant, tag, note}. The synthesizer groups by
|
||||
tag, counts distinct participants per tag, and ranks. This is the atomic-research discipline:
|
||||
an observation is evidence; an insight requires recurrence across independent sources.
|
||||
|
||||
Usage:
|
||||
python3 insight_synthesizer.py --sample
|
||||
python3 insight_synthesizer.py --input observations.json --min-sources 3
|
||||
python3 insight_synthesizer.py --input observations.json --output json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
import config_loader as _cfg
|
||||
except ImportError: # pragma: no cover
|
||||
_cfg = None
|
||||
|
||||
SAMPLE = {
|
||||
"study": "Onboarding discovery (mid-market HR)",
|
||||
"observations": [
|
||||
{"participant": "P1", "tag": "import-confusion", "note": "Couldn't find CSV import."},
|
||||
{"participant": "P2", "tag": "import-confusion", "note": "Expected import on the dashboard."},
|
||||
{"participant": "P3", "tag": "import-confusion", "note": "Gave up looking for bulk upload."},
|
||||
{"participant": "P1", "tag": "permissions-unclear", "note": "Unsure who could see reports."},
|
||||
{"participant": "P4", "tag": "permissions-unclear", "note": "Worried about data visibility."},
|
||||
{"participant": "P2", "tag": "wants-slack", "note": "Asked for a Slack integration."},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def synthesize(data: dict, min_sources: int) -> dict:
|
||||
obs = data.get("observations", [])
|
||||
by_tag_participants = defaultdict(set)
|
||||
by_tag_notes = defaultdict(list)
|
||||
for o in obs:
|
||||
tag = o.get("tag", "untagged")
|
||||
part = o.get("participant", "UNKNOWN")
|
||||
by_tag_participants[tag].add(part)
|
||||
by_tag_notes[tag].append({"participant": part, "note": o.get("note", "")})
|
||||
|
||||
candidates = []
|
||||
for tag, parts in by_tag_participants.items():
|
||||
n_sources = len(parts)
|
||||
is_insight = n_sources >= min_sources
|
||||
candidates.append({
|
||||
"tag": tag,
|
||||
"distinct_participants": n_sources,
|
||||
"observation_count": len(by_tag_notes[tag]),
|
||||
"classification": "INSIGHT" if is_insight else "ANECDOTE (single/low-source — do not generalize)",
|
||||
"evidence": by_tag_notes[tag],
|
||||
})
|
||||
candidates.sort(key=lambda c: (c["distinct_participants"], c["observation_count"]), reverse=True)
|
||||
|
||||
total_participants = len({o.get("participant") for o in obs})
|
||||
return {
|
||||
"study": data.get("study", "UNSPECIFIED"),
|
||||
"min_sources_for_insight": min_sources,
|
||||
"total_participants": total_participants,
|
||||
"candidates": candidates,
|
||||
"note": "An observation is evidence; an insight requires recurrence across independent participants. "
|
||||
"Anecdotes are surfaced, never promoted to insights.",
|
||||
}
|
||||
|
||||
|
||||
def _render_human(r: dict) -> str:
|
||||
lines = [f"Insight Synthesis: {r['study']}",
|
||||
f" total participants: {r['total_participants']} insight threshold: >= {r['min_sources_for_insight']} sources", ""]
|
||||
for c in r["candidates"]:
|
||||
lines.append(f"[{c['classification']}] {c['tag']} "
|
||||
f"({c['distinct_participants']} participants, {c['observation_count']} observations)")
|
||||
for e in c["evidence"]:
|
||||
lines.append(f" {e['participant']}: {e['note']}")
|
||||
lines.append("")
|
||||
lines.append(f"note: {r['note']}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description="Cluster coded observations into insights; flag anecdotes.")
|
||||
p.add_argument("--input", help="Path to JSON with observations[]")
|
||||
p.add_argument("--min-sources", type=int, default=None,
|
||||
help="min distinct participants to call it an insight (overrides onboarding)")
|
||||
p.add_argument("--output", choices=["human", "json"], default="human")
|
||||
p.add_argument("--sample", action="store_true", help="use the embedded sample")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
conf = _cfg.load_config() if _cfg else {}
|
||||
min_sources = args.min_sources if args.min_sources is not None else int(conf.get("insight_min_sources", 3))
|
||||
data = SAMPLE if (args.sample or not args.input) else json.load(open(args.input))
|
||||
result = synthesize(data, min_sources)
|
||||
if args.output == "json":
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(_render_human(result))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,124 @@
|
||||
#!/usr/bin/env python3
|
||||
"""onboard.py - Onboarding questionnaire for the product-research skill.
|
||||
|
||||
Stdlib-only. Asks the user a short set of questions BEFORE they plan a study, then
|
||||
writes the answers to a customization config read by every tool in this skill via
|
||||
config_loader.py. The answers become defaults for profile, the insight source-threshold,
|
||||
the default saturation method, and the high-stakes flag.
|
||||
|
||||
Modes: --show | --defaults | --set key=value (repeatable) | --reset | --scope {global,project}
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as _dt
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import config_loader as cfg # noqa: E402
|
||||
|
||||
INT_KEYS = {"insight_min_sources"}
|
||||
BOOL_KEYS = {"stakes_high"}
|
||||
|
||||
QUESTIONS = [
|
||||
("default_profile",
|
||||
"1. What kind of product is this?",
|
||||
["b2b-saas", "consumer-app", "enterprise", "marketplace", "hardware", "platform"], str),
|
||||
("insight_min_sources",
|
||||
"2. How many independent participants must support a finding before it counts as an insight (not an anecdote)?",
|
||||
None, int),
|
||||
("default_method",
|
||||
"3. Default sample-saturation method?",
|
||||
["usability", "thematic", "evaluative-coverage"], str),
|
||||
("stakes_high",
|
||||
"4. Is this high-stakes / high-heterogeneity research (raise sample sizes)?",
|
||||
["true", "false"], str),
|
||||
]
|
||||
|
||||
|
||||
def _coerce(key: str, value: str):
|
||||
if key in INT_KEYS:
|
||||
return int(value)
|
||||
if key in BOOL_KEYS:
|
||||
return str(value).strip().lower() in ("true", "yes", "y", "1")
|
||||
return value
|
||||
|
||||
|
||||
def _print_questions() -> None:
|
||||
print(f"Onboarding questions — {cfg.SKILL}:\n")
|
||||
for _k, prompt, choices, _c in QUESTIONS:
|
||||
line = f" {prompt}"
|
||||
if choices:
|
||||
line += f" [{' / '.join(choices)}]"
|
||||
print(line)
|
||||
|
||||
|
||||
def run_interactive(config: dict) -> dict:
|
||||
print(f"Onboarding — {cfg.SKILL}. Press Enter to keep the current/default value.\n")
|
||||
for key, prompt, choices, _caster in QUESTIONS:
|
||||
suffix = f" [{'/'.join(choices)}]" if choices else ""
|
||||
cur = f" (current: {config.get(key)})" if config.get(key) is not None else ""
|
||||
raw = input(f"{prompt}{suffix}{cur}: ").strip()
|
||||
if not raw:
|
||||
continue
|
||||
try:
|
||||
config[key] = _coerce(key, raw)
|
||||
except ValueError:
|
||||
print(f" ! invalid value for {key}, keeping current")
|
||||
return config
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description=f"Onboarding for the {cfg.SKILL} skill.")
|
||||
p.add_argument("--show", action="store_true")
|
||||
p.add_argument("--defaults", action="store_true", help="write built-in defaults, no prompt")
|
||||
p.add_argument("--set", action="append", default=[], metavar="key=value")
|
||||
p.add_argument("--reset", action="store_true")
|
||||
p.add_argument("--scope", choices=["global", "project"], default="global")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
if args.show:
|
||||
_print_questions()
|
||||
print("\nCurrent effective config:")
|
||||
print(json.dumps(cfg.load_config(), indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
if args.reset:
|
||||
path = cfg.project_config_path() if args.scope == "project" else cfg.GLOBAL_CONFIG_PATH
|
||||
if path.exists():
|
||||
path.unlink(); print(f"removed {path}")
|
||||
else:
|
||||
print(f"no config at {path}")
|
||||
return 0
|
||||
|
||||
config = cfg.load_config()
|
||||
|
||||
if args.set:
|
||||
for item in args.set:
|
||||
if "=" not in item:
|
||||
print(f"error: --set expects key=value, got '{item}'", file=sys.stderr)
|
||||
return 2
|
||||
k, v = item.split("=", 1)
|
||||
try:
|
||||
config[k] = _coerce(k, v)
|
||||
except ValueError:
|
||||
config[k] = v
|
||||
elif not args.defaults:
|
||||
if sys.stdin.isatty():
|
||||
config = run_interactive(config)
|
||||
else:
|
||||
print("non-interactive shell: use --defaults or --set key=value. Showing questions:\n")
|
||||
_print_questions()
|
||||
return 0
|
||||
|
||||
config["setup_completed_at"] = _dt.datetime.now(_dt.timezone.utc).isoformat()
|
||||
path = cfg.write_config(config, scope=args.scope)
|
||||
print(f"saved {cfg.SKILL} customization -> {path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,144 @@
|
||||
#!/usr/bin/env python3
|
||||
"""saturation_planner.py - Method-based participant/sample guidance with a confidence label.
|
||||
|
||||
Stdlib-only. Deterministic. NO LLM calls. NEVER fabricates insight: it gives method-based
|
||||
sample guidance and an explicit confidence level, surfacing limits.
|
||||
|
||||
Models:
|
||||
- usability (Nielsen): ~5 users per segment uncovers ~85% of problems at typical p=0.31;
|
||||
problems found = 1 - (1 - p)^n.
|
||||
- thematic saturation (Guest et al.): ~12 interviews per homogeneous group typically
|
||||
reaches saturation; >5 (Faulkner) when stakes/heterogeneity are high.
|
||||
- evaluative coverage: detectable-problem coverage for a chosen per-problem detection rate.
|
||||
|
||||
Usage:
|
||||
python3 saturation_planner.py --sample
|
||||
python3 saturation_planner.py --method usability --segments 2 --detection-rate 0.31
|
||||
python3 saturation_planner.py --method thematic --segments 3 --output json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
import config_loader as _cfg
|
||||
except ImportError: # pragma: no cover
|
||||
_cfg = None
|
||||
|
||||
METHODS = ["usability", "thematic", "evaluative-coverage"]
|
||||
|
||||
|
||||
def usability_plan(segments: int, p: float, target_coverage: float) -> dict:
|
||||
# n per segment to reach target coverage: n = ln(1 - target) / ln(1 - p)
|
||||
import math
|
||||
if not 0.0 < p < 1.0:
|
||||
raise ValueError("detection-rate must be in (0,1).")
|
||||
n = math.ceil(math.log(1 - target_coverage) / math.log(1 - p))
|
||||
coverage_at_5 = 1 - (1 - p) ** 5
|
||||
return {
|
||||
"method": "usability",
|
||||
"per_problem_detection_rate": p,
|
||||
"target_coverage": target_coverage,
|
||||
"n_per_segment": n,
|
||||
"segments": segments,
|
||||
"total_participants": n * segments,
|
||||
"coverage_at_5_per_segment": round(coverage_at_5, 3),
|
||||
"confidence": "MODERATE" if n >= 5 else "LOW (small-n usability finds problems, not rates)",
|
||||
"limits": "Usability tests surface problems, not their population prevalence. Do not report percentages.",
|
||||
}
|
||||
|
||||
|
||||
def thematic_plan(segments: int, stakes_high: bool) -> dict:
|
||||
base = 12 # Guest et al. typical saturation for a homogeneous group
|
||||
per_segment = base if not stakes_high else max(base, 15)
|
||||
return {
|
||||
"method": "thematic",
|
||||
"n_per_segment": per_segment,
|
||||
"segments": segments,
|
||||
"total_participants": per_segment * segments,
|
||||
"confidence": "MODERATE-HIGH" if per_segment >= 12 else "LOW",
|
||||
"limits": "Saturation is observed, not guaranteed; track new-theme rate and stop when it flattens. "
|
||||
"Faulkner (2003): more than 5 when heterogeneity or stakes are high.",
|
||||
}
|
||||
|
||||
|
||||
def evaluative_coverage_plan(segments: int, n_per_segment: int, p: float) -> dict:
|
||||
coverage = 1 - (1 - p) ** n_per_segment
|
||||
return {
|
||||
"method": "evaluative-coverage",
|
||||
"per_problem_detection_rate": p,
|
||||
"n_per_segment": n_per_segment,
|
||||
"segments": segments,
|
||||
"expected_problem_coverage": round(coverage, 3),
|
||||
"confidence": "MODERATE" if coverage >= 0.8 else "LOW",
|
||||
"limits": "Coverage is for the assumed detection rate; rarer problems need more participants.",
|
||||
}
|
||||
|
||||
|
||||
def plan(method: str, segments: int, p: float, target: float, stakes_high: bool, n: int) -> dict:
|
||||
if method == "usability":
|
||||
out = usability_plan(segments, p, target)
|
||||
elif method == "thematic":
|
||||
out = thematic_plan(segments, stakes_high)
|
||||
elif method == "evaluative-coverage":
|
||||
out = evaluative_coverage_plan(segments, n, p)
|
||||
else:
|
||||
raise ValueError(f"method must be one of {METHODS}.")
|
||||
out["disclaimer"] = "Method-based guidance with explicit confidence. This is not a power calculation; " \
|
||||
"it never claims an insight the data cannot support."
|
||||
return out
|
||||
|
||||
|
||||
def _render_human(r: dict) -> str:
|
||||
lines = [f"Saturation / Sample Plan (method: {r['method']})", ""]
|
||||
for k, v in r.items():
|
||||
if k in ("method",):
|
||||
continue
|
||||
lines.append(f" {k:32s} : {v}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description="Method-based product-research sample guidance with confidence.")
|
||||
p.add_argument("--method", choices=METHODS, default=None, help="overrides onboarding default_method")
|
||||
p.add_argument("--segments", type=int, default=1)
|
||||
p.add_argument("--detection-rate", type=float, default=0.31, help="per-problem detection rate (usability)")
|
||||
p.add_argument("--target-coverage", type=float, default=0.85, help="target problem coverage (usability)")
|
||||
p.add_argument("--stakes-high", action="store_true", help="raise thematic n for high heterogeneity/stakes")
|
||||
p.add_argument("--n-per-segment", type=int, default=8, help="n per segment (evaluative-coverage)")
|
||||
p.add_argument("--output", choices=["human", "json"], default="human")
|
||||
p.add_argument("--sample", action="store_true", help="use the embedded sample")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
conf = _cfg.load_config() if _cfg else {}
|
||||
method = args.method or conf.get("default_method", "usability")
|
||||
stakes_high = args.stakes_high or bool(conf.get("stakes_high", False))
|
||||
|
||||
if args.sample:
|
||||
try:
|
||||
result = plan("usability", 2, 0.31, 0.85, False, 8)
|
||||
except ValueError as e:
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 2
|
||||
else:
|
||||
try:
|
||||
result = plan(method, args.segments, args.detection_rate,
|
||||
args.target_coverage, stakes_high, args.n_per_segment)
|
||||
except ValueError as e:
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
if args.output == "json":
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(_render_human(result))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,139 @@
|
||||
#!/usr/bin/env python3
|
||||
"""study_designer.py - Select a product-research method from goal + stage, emit a plan skeleton.
|
||||
|
||||
Stdlib-only. Deterministic. NO LLM calls.
|
||||
|
||||
Maps (research goal x product stage) to an appropriate method and emits a method-matched
|
||||
plan skeleton (objective framing, participant criteria, task/guide structure, success
|
||||
criteria). The core discipline: GENERATIVE goals (discover problems) and EVALUATIVE goals
|
||||
(test a solution) demand different methods — picking the wrong one is the most common error.
|
||||
|
||||
Usage:
|
||||
python3 study_designer.py --sample
|
||||
python3 study_designer.py --goal discovery --stage concept --profile b2b-saas
|
||||
python3 study_designer.py --goal evaluative --stage live --output json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
import config_loader as _cfg
|
||||
except ImportError: # pragma: no cover
|
||||
_cfg = None
|
||||
|
||||
PROFILES = ["b2b-saas", "consumer-app", "enterprise", "marketplace", "hardware", "platform"]
|
||||
|
||||
# (goal, stage) -> method. goal in {discovery, evaluative, validation}; stage in {concept, prototype, beta, live}
|
||||
METHOD_MAP = {
|
||||
("discovery", "concept"): "generative interviews (semi-structured)",
|
||||
("discovery", "prototype"): "contextual inquiry",
|
||||
("discovery", "beta"): "diary study + follow-up interviews",
|
||||
("discovery", "live"): "behavioral analytics review + generative interviews",
|
||||
("evaluative", "concept"): "concept test (comprehension + desirability)",
|
||||
("evaluative", "prototype"): "moderated usability test",
|
||||
("evaluative", "beta"): "unmoderated usability test + task-success metrics",
|
||||
("evaluative", "live"): "benchmark usability study (SUS / task time)",
|
||||
("validation", "concept"): "survey (desirability + willingness signals)",
|
||||
("validation", "prototype"): "prototype A/B preference test",
|
||||
("validation", "beta"): "fake-door / feature-demand test",
|
||||
("validation", "live"): "live A/B experiment (route to product-team/experiment-designer)",
|
||||
}
|
||||
|
||||
GUIDE_SKELETONS = {
|
||||
"generative": ["Warm-up + context", "Recent relevant experience (story, not opinion)",
|
||||
"Workarounds + frustrations", "Jobs-to-be-done probe", "Magic-wand / wrap"],
|
||||
"evaluative": ["Pre-task context", "Task 1 (representative)", "Task 2 (edge)",
|
||||
"Observation: where do they hesitate/err?", "Post-task SUS / debrief"],
|
||||
"validation": ["Screener", "Stimulus exposure", "Comprehension + desirability items",
|
||||
"Trade-off / preference items", "Behavioral-intent item"],
|
||||
}
|
||||
|
||||
|
||||
def design(goal: str, stage: str, profile: str) -> dict:
|
||||
if profile not in PROFILES:
|
||||
raise ValueError(f"Unknown profile '{profile}'. Choose from {PROFILES}.")
|
||||
key = (goal, stage)
|
||||
if key not in METHOD_MAP:
|
||||
raise ValueError(f"No method for goal={goal}, stage={stage}. "
|
||||
f"goal in [discovery,evaluative,validation]; stage in [concept,prototype,beta,live].")
|
||||
method = METHOD_MAP[key]
|
||||
family = "generative" if goal == "discovery" else ("evaluative" if goal == "evaluative" else "validation")
|
||||
redirect = None
|
||||
if "experiment-designer" in method:
|
||||
redirect = "Live A/B is a product experiment — use product-team/experiment-designer, not this skill."
|
||||
return {
|
||||
"goal": goal,
|
||||
"stage": stage,
|
||||
"profile": profile,
|
||||
"method": method,
|
||||
"method_family": family,
|
||||
"objective_framing": f"A {family} study at the {stage} stage to {('discover unmet needs' if family=='generative' else 'evaluate the solution' if family=='evaluative' else 'validate demand/desirability')}.",
|
||||
"participant_criteria": [
|
||||
"Recruit to the target segment (screen for the job, not a job title).",
|
||||
"Exclude internal/biased participants and prior-study repeats unless longitudinal.",
|
||||
"Recruit per-segment if results will be reported per-segment.",
|
||||
],
|
||||
"guide_skeleton": GUIDE_SKELETONS[family],
|
||||
"success_criteria": [
|
||||
"Generative: themes recur across independent participants (saturation).",
|
||||
"Evaluative: task-success rate + severity-rated problem list.",
|
||||
"Validation: pre-registered desirability / preference threshold.",
|
||||
],
|
||||
"redirect": redirect,
|
||||
"note": "Method must match the goal. A usability test cannot discover unmet needs; an interview cannot measure task success.",
|
||||
}
|
||||
|
||||
|
||||
def _render_human(r: dict) -> str:
|
||||
lines = [f"Study Design: goal={r['goal']}, stage={r['stage']}, profile={r['profile']}", "",
|
||||
f" Recommended method: {r['method']} (family: {r['method_family']})",
|
||||
f" Objective: {r['objective_framing']}", "", " Participant criteria:"]
|
||||
for c in r["participant_criteria"]:
|
||||
lines.append(f" - {c}")
|
||||
lines.append(" Guide skeleton:")
|
||||
for i, g in enumerate(r["guide_skeleton"], 1):
|
||||
lines.append(f" {i}. {g}")
|
||||
lines.append(" Success criteria:")
|
||||
for s in r["success_criteria"]:
|
||||
lines.append(f" - {s}")
|
||||
if r["redirect"]:
|
||||
lines += ["", f" !! {r['redirect']}"]
|
||||
lines += ["", f"note: {r['note']}"]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description="Select a product-research method from goal + stage.")
|
||||
p.add_argument("--goal", choices=["discovery", "evaluative", "validation"], default="discovery")
|
||||
p.add_argument("--stage", choices=["concept", "prototype", "beta", "live"], default="prototype")
|
||||
p.add_argument("--profile", default=None, choices=PROFILES,
|
||||
help="overrides onboarding default_profile")
|
||||
p.add_argument("--output", choices=["human", "json"], default="human")
|
||||
p.add_argument("--sample", action="store_true", help="use the embedded sample")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
conf = _cfg.load_config() if _cfg else {}
|
||||
profile_default = conf.get("default_profile", "b2b-saas")
|
||||
goal, stage, profile = ("discovery", "prototype", profile_default) if args.sample \
|
||||
else (args.goal, args.stage, args.profile or profile_default)
|
||||
try:
|
||||
result = design(goal, stage, profile)
|
||||
except ValueError as e:
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
if args.output == "json":
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(_render_human(result))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,146 @@
|
||||
---
|
||||
name: research-finance
|
||||
description: Use when managing the money for an internal R&D program or portfolio — building a multi-period program budget with the F&A (indirect) split, tracking burn rate and runway against value-inflection milestones, or routing R&D cost items to a capitalize-vs-expense determination. Every budget output surfaces its assumptions block; capitalize-vs-expense is decision-support only and routes to a named finance owner — it never books an entry or decides accounting treatment. Distinct from finance/financial-analysis (corporate DCF, close, valuation) and research/grants (funding discovery — this manages money already won).
|
||||
version: 2.9.0
|
||||
author: claude-code-skills
|
||||
license: MIT
|
||||
tags: [research-ops, research-finance, rd-budget, burn-rate, runway, fa-rate, capitalize-vs-expense, portfolio]
|
||||
compatible_tools: [claude-code, codex-cli, cursor, antigravity, opencode, gemini-cli]
|
||||
---
|
||||
|
||||
# research-finance
|
||||
|
||||
Financial management of internal R&D programs and portfolios: program budgeting with F&A, burn/runway tracking, and capitalize-vs-expense routing. Every number ships with its **assumptions block**, and accounting-treatment calls **route to a named finance owner** — this skill never books an entry.
|
||||
|
||||
## Purpose
|
||||
|
||||
R&D finance partners, program controllers, and operations leads manage money that has already been allocated or raised — not the corporate close, not the next funding round, not finding a grant. This skill structures three recurring decisions:
|
||||
|
||||
Three deterministic tools:
|
||||
|
||||
1. `program_budget_planner.py` — Builds a multi-period budget from work-package line items, applies the F&A (indirect) rate to an MTDC-style eligible base, and rolls up direct / F&A / fully-loaded cost per period with an explicit assumptions block.
|
||||
2. `burn_runway_tracker.py` — Computes average + trailing burn, runway in periods/months, and whether each value-inflection milestone is reachable before cash runs out. Flags accelerating burn and below-threshold runway.
|
||||
3. `capex_vs_opex_router.py` — Scores each R&D cost item against the IAS 38 development-phase criteria (or flags US GAAP ASC 730 expense-as-incurred) and routes it to **CAPITALIZE-CANDIDATE / EXPENSE / FINANCE-OWNER-REVIEW** with a named owner. Never auto-decides.
|
||||
|
||||
## When to use
|
||||
|
||||
Invoke this skill when:
|
||||
|
||||
- You are building or revising an R&D program budget and need the F&A split made explicit.
|
||||
- A program's runway is in question and you need a milestone-vs-cash read.
|
||||
- Finance asks whether a development cost can be capitalized and you need a defensible first routing.
|
||||
- You are preparing a portfolio review and need per-program burn consistency.
|
||||
|
||||
**Do NOT use this skill to**: run corporate DCF / valuation / close (use `finance/financial-analysis`), discover or position grants (use `research/grants`), or make the final accounting determination (that is the controller's + auditor's call — this tool only routes).
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Lay out the program** — Fill `assets/rd_program_budget_template.md` with work-package lines, categories, and per-period amounts.
|
||||
2. **Build the budget** — Run `program_budget_planner.py --input program.json --profile {pharma-rd|biotech|medtech|deep-tech|software-rd|university-lab} --fa-rate <negotiated rate>`. Read direct / F&A / fully-loaded rollups + assumptions.
|
||||
3. **Track burn & runway** — Run `burn_runway_tracker.py --input ledger.json --threshold-months 6`. Read runway + milestone verdicts + flags.
|
||||
4. **Route accounting treatment** — Run `capex_vs_opex_router.py --input costs.json --standard {ifrs|usgaap}`. Read the per-item routing; send CAPITALIZE-CANDIDATE and FINANCE-OWNER-REVIEW items to the named owner.
|
||||
5. **Assemble the review** — Combine into a program-finance packet. Every number carries its assumptions; treatment calls carry a named owner.
|
||||
|
||||
## Scripts
|
||||
|
||||
| Script | Purpose | Profiles |
|
||||
|---|---|---|
|
||||
| `scripts/program_budget_planner.py` | Multi-period budget + F&A split + assumptions | pharma-rd, biotech, medtech, deep-tech, software-rd, university-lab |
|
||||
| `scripts/burn_runway_tracker.py` | Burn, runway, milestone-vs-cash alignment | n/a (ledger-driven) |
|
||||
| `scripts/capex_vs_opex_router.py` | IAS 38 / ASC 730 routing to named finance owner | pharma-rd, biotech, medtech, deep-tech, software-rd, university-lab |
|
||||
|
||||
All three: stdlib-only, `--help`, `--sample`, `--output {human,json}`.
|
||||
|
||||
## Onboarding & customization
|
||||
|
||||
Run the onboarding questionnaire **once before you start** — it captures your defaults so every tool in this skill is pre-configured. Customization is the point: the answers actually change tool behavior.
|
||||
|
||||
```bash
|
||||
python3 scripts/onboard.py # interactive (also: --defaults, --set key=value, --reset)
|
||||
python3 scripts/onboard.py --show # see the questions + current effective config
|
||||
```
|
||||
|
||||
Answers are saved to `~/.config/research-ops/research-finance.json` (global) or `./.research-ops/research-finance.json` (`--scope project`) and are read automatically by `config_loader.py`. They set the default R&D-area **profile**, the default **F&A rate**, the **runway alert threshold**, the **accounting standard**, and the named **finance owner** printed on capitalize-vs-expense routing. CLI flags always override saved config; `RESEARCH_OPS_NO_CONFIG=1` ignores it.
|
||||
|
||||
**The five questions:** R&D area · F&A rate · runway threshold · accounting standard · finance owner.
|
||||
|
||||
## Optimize with autoresearch (opt-in)
|
||||
|
||||
This skill ships an **isolated, opt-in** bridge to `engineering/autoresearch-agent`. Only when you ask to "optimize" / "extend runway" / "run a loop" does an autoresearch experiment iteratively improve a program plan against this skill's runway metric. `scripts/ar_evaluator.py` is the ground-truth evaluator; it prints `runway_months: <float>` (higher is better).
|
||||
|
||||
```bash
|
||||
/ar:setup --domain custom --name extend-runway \
|
||||
--target ledger.json \
|
||||
--eval "python3 ar_evaluator.py --target ledger.json" \
|
||||
--metric runway_months --direction higher
|
||||
/ar:loop custom/extend-runway
|
||||
```
|
||||
|
||||
Isolated: no hard dependency — autoresearch runs only on demand, and the loop edits `ledger.json`, never the evaluator.
|
||||
|
||||
## References
|
||||
|
||||
- `references/rd_program_finance_canon.md` — IAS 38 (research vs development); ASC 730 + ASC 985-20; Uniform Guidance 2 CFR 200 (F&A); FASB/IFRS capitalization criteria; NICRA basics.
|
||||
- `references/burn_and_portfolio.md` — Cooper stage-gate; rNPV / real-options for R&D; risk-adjusted portfolio ROI; burn-rate / runway frameworks; milestone-based budgeting.
|
||||
- `references/indirect_rate_modeling.md` — F&A pool composition (facilities + administration); MTDC base; de minimis 10%; fringe/overhead loading; CAS primer.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- The F&A rate is the most error-prone input. The planner applies whatever rate you pass; it warns you to confirm it is a negotiated NICRA, not a guess.
|
||||
- Burn/runway uses the trailing (recent-weighted) burn as the forward run-rate and assumes flat forward spend unless your ledger encodes a ramp.
|
||||
- The capex router asserts criteria from your input; asserting "technical feasibility" does not make it true — the named finance owner and auditor validate it.
|
||||
- Profiles annotate context (e.g., "most drug R&D is expensed") but do not change the accounting test.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Stating a budget number without its assumptions.** F&A rate, escalation, and base must travel with the number.
|
||||
- **Auto-deciding capitalize-vs-expense.** This tool routes; the controller (and auditor where required) decides.
|
||||
- **Using lifetime-average burn for runway.** Recent burn is the honest forward run-rate; averages hide a slowdown or a ramp.
|
||||
- **Applying F&A to the full base.** Capital equipment, large subaward portions, and certain categories are MTDC-exempt.
|
||||
- **Confusing this with corporate finance.** Valuation, close, and fundraising live in `finance/`.
|
||||
|
||||
## Distinct from
|
||||
|
||||
| Sibling / neighbor | Scope | Difference |
|
||||
|---|---|---|
|
||||
| `finance/financial-analysis` | Corporate DCF, ratios, close, rolling forecast, SaaS metrics | That is **company-level**; this is **R&D-program-level** |
|
||||
| `research/grants` | NIH funding discovery + positioning | That **finds funding**; this **manages money already won** |
|
||||
| `clinical-research` (sibling) | Study design + feasibility + budget gate-check | That **scopes** the study; this **funds + tracks** the program |
|
||||
| `ra-qm-team` | Regulatory/QM submission | Unrelated — no financial scope |
|
||||
|
||||
## Quick examples
|
||||
|
||||
```bash
|
||||
python3 scripts/program_budget_planner.py --sample
|
||||
python3 scripts/program_budget_planner.py --input program.json --profile university-lab --fa-rate 0.585
|
||||
python3 scripts/burn_runway_tracker.py --sample --output json
|
||||
python3 scripts/capex_vs_opex_router.py --sample --standard ifrs
|
||||
```
|
||||
|
||||
The sample budget excludes the sequencer (capital equipment) and CRO subaward from the F&A base; the capex router routes exploratory screening to EXPENSE, a fully-criteria'd pilot line to CAPITALIZE-CANDIDATE, and a partial-criteria software build to FINANCE-OWNER-REVIEW.
|
||||
|
||||
## Forcing-question library (Matt Pocock grill discipline)
|
||||
|
||||
Walked one at a time by `/cs:grill-research-ops` or the orchestrator. Recommended answer + canon citation per question. Never bundled.
|
||||
|
||||
1. **"Is this spend in the research phase or the development phase — and can you evidence technical feasibility?"**
|
||||
Recommended: research = expense; development = capitalize-candidate only with feasibility evidence, routed to a named finance owner.
|
||||
Canon: IAS 38.54-57; ASC 730.
|
||||
|
||||
2. **"What F&A / indirect rate are you applying, and is it your negotiated NICRA, a de minimis 10%, or an assumption?"**
|
||||
Recommended: use the negotiated rate; if assumed, flag it explicitly.
|
||||
Canon: 2 CFR 200 (Uniform Guidance); NICRA basics.
|
||||
|
||||
3. **"What's runway in months at current burn, and does it clear the next value-inflection milestone?"**
|
||||
Recommended: runway must cover the milestone plus a buffer; surface the gap.
|
||||
Canon: Cooper stage-gate; SaaS/startup efficiency frameworks (a16z, Bessemer).
|
||||
|
||||
4. **"Is portfolio ROI risk-adjusted (rNPV / probability-of-success weighted) or raw NPV?"**
|
||||
Recommended: risk-adjusted; raw NPV overstates R&D value.
|
||||
Canon: rNPV drug-development valuation; real-options literature.
|
||||
|
||||
5. **"Who is the named finance / controller owner who signs the capitalize-vs-expense treatment?"**
|
||||
Recommended: name them — this tool recommends, it never books the entry.
|
||||
Canon: ASC 730 / IAS 38 governance; auditor sign-off requirements.
|
||||
|
||||
Walk depth-first. Lock 1-2 before opening 3-5. After all are answered, invoke `program_budget_planner.py` → `burn_runway_tracker.py` → `capex_vs_opex_router.py`.
|
||||
@@ -0,0 +1,48 @@
|
||||
# R&D Program Budget — Template
|
||||
|
||||
> Fill this before running `program_budget_planner.py`. Every number must travel with its
|
||||
> assumptions. Capitalize-vs-expense calls route to a named finance owner — this is not the
|
||||
> place to decide accounting treatment.
|
||||
|
||||
## 1. Program identification
|
||||
- Program name:
|
||||
- R&D area / profile: [pharma-rd | biotech | medtech | deep-tech | software-rd | university-lab]
|
||||
- Number of periods + period label (month / quarter / year):
|
||||
- Funding source(s):
|
||||
|
||||
## 2. F&A (indirect) basis
|
||||
- F&A rate applied: ____%
|
||||
- Rate type: [negotiated NICRA | de minimis 10% | internal assumption — FLAG IT]
|
||||
- Fringe rate (loaded onto salaries before F&A): ____%
|
||||
|
||||
## 3. Work packages (per-period amounts)
|
||||
| Work package | Category | F&A-eligible? | P1 | P2 | P3 | P4 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Personnel (FTEs) | personnel | yes | | | | |
|
||||
| Consumables / supplies | supplies | yes | | | | |
|
||||
| Capital equipment | capital_equipment | NO (MTDC-exempt) | | | | |
|
||||
| Subaward / CRO (>$25k) | subaward_over_25k | NO (over $25k exempt) | | | | |
|
||||
| Travel | travel | yes | | | | |
|
||||
|
||||
> Categories that are MTDC-exempt: capital_equipment, subaward_over_25k, tuition, patient_care.
|
||||
|
||||
## 4. Milestones (for burn/runway)
|
||||
| Milestone | Periods from now | Cumulative cash needed |
|
||||
|---|---|---|
|
||||
| | | |
|
||||
|
||||
## 5. Capitalize-vs-expense candidates (for routing only)
|
||||
| Cost item | Phase (research / development / software-development) | Standard (ifrs / usgaap) |
|
||||
|---|---|---|
|
||||
| | | |
|
||||
|
||||
## 6. Assumptions register
|
||||
- F&A rate basis:
|
||||
- Escalation assumption:
|
||||
- Forward burn assumption (flat / ramp):
|
||||
- Probability-of-success weighting (for any portfolio ROI):
|
||||
|
||||
## 7. Named owners
|
||||
- R&D Finance Controller:
|
||||
- External Auditor (if capitalization in play):
|
||||
- Program Lead:
|
||||
@@ -0,0 +1,28 @@
|
||||
# Burn, Runway, and R&D Portfolio Management
|
||||
|
||||
Reference for burn/runway tracking and risk-adjusted portfolio decisions. Pairs with `burn_runway_tracker.py`.
|
||||
|
||||
## Burn and runway done honestly
|
||||
|
||||
**Burn rate** is cash spent per period; **runway** is cash-on-hand ÷ forward run-rate. The honest forward run-rate is the **trailing** (recent-weighted) burn, not the lifetime average — averages mask both an accelerating spend and a funded ramp. The tracker uses trailing burn and flags when trailing exceeds 115% of the lifetime average (an acceleration signal). Runway must be measured against **value-inflection milestones**: cash that runs out one month before analytical validation is materially worse than the same runway that clears it, because reaching the milestone changes the program's financing options and valuation.
|
||||
|
||||
## Stage-gate portfolio management
|
||||
|
||||
Robert Cooper's **Stage-Gate** model structures R&D as a sequence of stages separated by go/kill **gates**. Each gate is a real-options decision: spend the next tranche, or kill and redeploy. The discipline is that money is committed one stage at a time, against pre-defined criteria — not as a lump sum at kickoff. This is why milestone-vs-cash alignment is the core runway question.
|
||||
|
||||
## Risk-adjusted valuation
|
||||
|
||||
Raw NPV systematically overstates R&D value because it ignores attrition. **Risk-adjusted NPV (rNPV)** weights each phase's cash flows by the cumulative probability of success of reaching it — in drug development, the product of per-phase success rates (which compound to single-digit percentages from preclinical to approval). **Real-options** valuation goes further, pricing the optionality of being able to abandon. For portfolio ROI, always state whether the number is raw NPV or risk-adjusted; the difference is often an order of magnitude.
|
||||
|
||||
## Efficiency benchmarks
|
||||
|
||||
Startup/SaaS efficiency frameworks (a16z's burn multiple, Bessemer's efficiency score) translate to R&D portfolios as "value created per dollar burned." They are blunt but useful for cross-program comparison when paired with milestone progress.
|
||||
|
||||
## Sources
|
||||
|
||||
1. Cooper, R.G., *Winning at New Products: Creating Value Through Innovation*, 5th ed. (2017) — Stage-Gate.
|
||||
2. Stewart, Allison & Johnson, *Putting a price on biotechnology* — Nature Biotechnology 2001 (rNPV in drug development).
|
||||
3. Trigeorgis, L., *Real Options: Managerial Flexibility and Strategy in Resource Allocation* (MIT Press).
|
||||
4. DiMasi, Grabowski & Hansen, *Innovation in the pharmaceutical industry: New estimates of R&D costs* — J Health Econ 2016 (attrition / phase success rates).
|
||||
5. a16z, *The burn multiple* and Bessemer State of the Cloud efficiency benchmarks.
|
||||
6. Chan & Thornhill, *R&D portfolio management* — R&D Management literature.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Indirect (F&A) Rate Modeling
|
||||
|
||||
Deep reference for the F&A rate — the single most error-prone input in an R&D budget. Pairs with `program_budget_planner.py`.
|
||||
|
||||
## What the F&A rate actually is
|
||||
|
||||
The F&A rate recovers shared costs that cannot be traced to a single program. It is composed of two pools:
|
||||
|
||||
- **Facilities** — depreciation on buildings and equipment, interest on facility debt, operations & maintenance, library, utilities.
|
||||
- **Administration** — general administration, departmental administration, sponsored-projects administration, student services (in universities).
|
||||
|
||||
The rate is computed as (indirect pool ÷ allocation base) and applied to that base on each program.
|
||||
|
||||
## The base matters as much as the rate
|
||||
|
||||
A 55% rate on a $1M total budget is *not* $550k of F&A — because the rate applies only to the **MTDC base**, which excludes:
|
||||
|
||||
- Capital equipment (typically items > $5,000 with > 1-year life)
|
||||
- The portion of **each** subaward exceeding $25,000 (the first $25k is in the base; the rest is exempt)
|
||||
- Tuition remission
|
||||
- Patient-care costs
|
||||
- Rental of off-site facilities, scholarships, participant support
|
||||
|
||||
So a budget heavy in equipment and large subawards has a much smaller F&A base than its headline total. The planner models this exclusion explicitly.
|
||||
|
||||
## Negotiated vs de minimis
|
||||
|
||||
- **NICRA** — the Negotiated Indirect Cost Rate Agreement, established with a cognizant federal agency. This is the authoritative rate for federally funded work.
|
||||
- **De minimis 10%** — under 2 CFR 200.414(f), an entity that has never had a negotiated rate may elect a flat 10% of MTDC. Simpler, almost always lower than a negotiated research rate.
|
||||
|
||||
## Fringe and the loading stack
|
||||
|
||||
Personnel costs load in layers: base salary → **fringe** (benefits, often 25-35%) → then F&A applies to salary+fringe (both are in the MTDC base). Modeling fringe separately from F&A avoids double counting or under-recovery.
|
||||
|
||||
## Sources
|
||||
|
||||
1. 2 CFR 200.414, *Indirect (F&A) costs*, and Appendix III (IHEs) / Appendix IV (nonprofits).
|
||||
2. 2 CFR 200.1, definition of *Modified Total Direct Cost (MTDC)*.
|
||||
3. NIH Grants Policy Statement, indirect-cost chapter; DHHS Cost Allocation Services NICRA guidance.
|
||||
4. Cost Accounting Standards Board, 48 CFR 9904 (CAS 410, 418 on allocation).
|
||||
5. COGR (Council on Governmental Relations), *Indirect Cost / F&A* primers and white papers.
|
||||
6. Federal Demonstration Partnership materials on subaward and MTDC treatment.
|
||||
@@ -0,0 +1,29 @@
|
||||
# R&D Program Finance Canon
|
||||
|
||||
Reference for the accounting and budgeting rules that govern internal R&D spend. Pairs with `program_budget_planner.py` and `capex_vs_opex_router.py`.
|
||||
|
||||
## The central question: research vs development
|
||||
|
||||
The accounting treatment of R&D hinges on a phase distinction that the two major frameworks handle differently:
|
||||
|
||||
- **IFRS (IAS 38)** — *Research* costs are always **expensed**. *Development* costs **must be capitalized** once all six conditions are met: (1) technical feasibility, (2) intention to complete, (3) ability to use or sell, (4) probable future economic benefit, (5) adequate resources to complete, (6) reliable measurement of expenditure. This is not optional under IFRS — if the criteria are met, capitalization is required.
|
||||
- **US GAAP (ASC 730)** — R&D is **expensed as incurred**, full stop, with narrow exceptions. The main exception is software: **ASC 985-20** (software to be sold) capitalizes costs after *technological feasibility*; **ASC 350-40** (internal-use software) capitalizes during the application-development stage.
|
||||
|
||||
This divergence is why the router takes a `--standard {ifrs,usgaap}` flag: the same cost item can be EXPENSE under US GAAP and CAPITALIZE-CANDIDATE under IFRS.
|
||||
|
||||
## F&A / indirect cost (the budgeting half)
|
||||
|
||||
Direct costs are traceable to the program (personnel, supplies). **Facilities & Administrative (F&A)**, a.k.a. indirect or overhead, covers shared costs (building, utilities, administration). For federally funded research, F&A is governed by **Uniform Guidance (2 CFR 200)**: organizations negotiate a rate (the NICRA — Negotiated Indirect Cost Rate Agreement) or use the **de minimis 10%** rate. F&A applies to the **Modified Total Direct Cost (MTDC)** base, which *excludes* capital equipment, the portion of each subaward over $25,000, tuition, and patient-care costs. The budget planner enforces this MTDC exclusion.
|
||||
|
||||
## Why disclosure matters
|
||||
|
||||
A budget number is only as trustworthy as its rate basis and escalation assumption. Two budgets for the same program can differ 40%+ purely on the F&A rate and the base. Every output of the planner ships an assumptions block for exactly this reason.
|
||||
|
||||
## Sources
|
||||
|
||||
1. IAS 38, *Intangible Assets* — IASB (research vs development, paragraphs 54-67).
|
||||
2. FASB ASC 730, *Research and Development*; ASC 985-20, *Software — Costs of Software to Be Sold, Leased, or Marketed*; ASC 350-40, *Internal-Use Software*.
|
||||
3. 2 CFR 200 (Uniform Guidance), Subpart E — Cost Principles, esp. §200.414 (Indirect F&A costs) and the MTDC definition (§200.1).
|
||||
4. Cost Accounting Standards (CAS), 48 CFR 9904 — for federally funded R&D contractors.
|
||||
5. KPMG / PwC / Deloitte IFRS-vs-US-GAAP comparison guides (R&D and intangibles chapters).
|
||||
6. AICPA Accounting & Valuation Guide, *Research and Development*.
|
||||
@@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env python3
|
||||
"""ar_evaluator.py - Autoresearch evaluator for the research-finance skill (OPT-IN).
|
||||
|
||||
Stdlib-only. The ISOLATED bridge to engineering/autoresearch-agent. It does NOT call
|
||||
autoresearch; it is the ground-truth evaluator an autoresearch loop runs after editing
|
||||
the target ledger/budget. It reads a ledger JSON, computes runway via burn_runway_tracker,
|
||||
and prints ONE metric line:
|
||||
|
||||
runway_months: <float> (higher is better)
|
||||
|
||||
Optimize a program plan to maximize runway (e.g., resequencing spend) while the agent
|
||||
edits the target. The user opts in explicitly:
|
||||
/ar:setup --domain custom --name extend-runway \\
|
||||
--target ledger.json --eval "python3 ar_evaluator.py --target ledger.json" \\
|
||||
--metric runway_months --direction higher
|
||||
|
||||
Direct use:
|
||||
python3 ar_evaluator.py --sample
|
||||
python3 ar_evaluator.py --target ledger.json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import config_loader as cfg # noqa: E402
|
||||
import burn_runway_tracker as brt # noqa: E402
|
||||
|
||||
METRIC = "runway_months"
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
c = cfg.load_config()
|
||||
p = argparse.ArgumentParser(description="Autoresearch evaluator: R&D program runway in months.")
|
||||
p.add_argument("--target", help="path to ledger JSON (or env AR_TARGET)")
|
||||
p.add_argument("--threshold-months", type=float, default=None)
|
||||
p.add_argument("--sample", action="store_true")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
threshold = args.threshold_months if args.threshold_months is not None \
|
||||
else c.get("runway_threshold_months", 6)
|
||||
|
||||
if args.sample:
|
||||
data = brt.SAMPLE
|
||||
else:
|
||||
target = args.target or os.environ.get("AR_TARGET")
|
||||
if not target:
|
||||
print("error: provide --target <ledger.json> or set AR_TARGET", file=sys.stderr)
|
||||
return 2
|
||||
try:
|
||||
with open(target) as f:
|
||||
data = json.load(f)
|
||||
except (OSError, json.JSONDecodeError) as e:
|
||||
print(f"{METRIC}: N/A")
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
try:
|
||||
result = brt.analyze(data, threshold)
|
||||
except ValueError as e:
|
||||
print(f"{METRIC}: N/A")
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"{METRIC}: {result['runway_months_approx']}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,153 @@
|
||||
#!/usr/bin/env python3
|
||||
"""burn_runway_tracker.py - Compute R&D program burn, runway, and milestone-vs-cash alignment.
|
||||
|
||||
Stdlib-only. Deterministic. NO LLM calls. Surfaces the assumption behind every number.
|
||||
|
||||
Given cash-on-hand, a period ledger of actual spend, and upcoming milestones (each with a
|
||||
period index and the cash needed to reach it), computes:
|
||||
- average + trailing burn rate
|
||||
- runway in periods and (approx) months
|
||||
- whether each value-inflection milestone is reachable before cash runs out
|
||||
|
||||
Usage:
|
||||
python3 burn_runway_tracker.py --sample
|
||||
python3 burn_runway_tracker.py --input ledger.json --threshold-months 6
|
||||
python3 burn_runway_tracker.py --input ledger.json --output json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
import config_loader as _cfg
|
||||
except ImportError: # pragma: no cover
|
||||
_cfg = None
|
||||
|
||||
SAMPLE = {
|
||||
"program": "Next-Gen Assay Platform",
|
||||
"cash_on_hand": 3200000,
|
||||
"period_label": "month",
|
||||
"actual_spend": [285000, 305000, 330000, 360000],
|
||||
"milestones": [
|
||||
{"name": "Analytical validation", "period_from_now": 3, "cumulative_cash_needed": 1000000},
|
||||
{"name": "First-in-human readiness", "period_from_now": 9, "cumulative_cash_needed": 3400000},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def analyze(data: dict, threshold_months: float) -> dict:
|
||||
spend = [float(x) for x in data.get("actual_spend", [])]
|
||||
cash = float(data.get("cash_on_hand", 0.0))
|
||||
label = data.get("period_label", "month")
|
||||
months_per_period = 1.0 if label == "month" else (3.0 if label == "quarter" else 1.0)
|
||||
|
||||
if not spend:
|
||||
raise ValueError("actual_spend must contain at least one period.")
|
||||
|
||||
avg_burn = sum(spend) / len(spend)
|
||||
trailing_n = min(3, len(spend))
|
||||
trailing_burn = sum(spend[-trailing_n:]) / trailing_n
|
||||
# Use trailing burn (more recent) as the forward run-rate.
|
||||
run_rate = trailing_burn if trailing_burn > 0 else avg_burn
|
||||
runway_periods = cash / run_rate if run_rate > 0 else float("inf")
|
||||
runway_months = runway_periods * months_per_period
|
||||
|
||||
milestones_out = []
|
||||
for m in data.get("milestones", []):
|
||||
needed = float(m.get("cumulative_cash_needed", 0.0))
|
||||
period_from_now = float(m.get("period_from_now", 0))
|
||||
reachable_cash = needed <= cash
|
||||
reachable_time = period_from_now <= runway_periods
|
||||
verdict = "REACHABLE" if (reachable_cash and reachable_time) else "AT-RISK"
|
||||
milestones_out.append({
|
||||
"name": m.get("name", "UNNAMED"),
|
||||
"period_from_now": period_from_now,
|
||||
"cumulative_cash_needed": needed,
|
||||
"cash_covers": reachable_cash,
|
||||
"runway_covers_timing": reachable_time,
|
||||
"verdict": verdict,
|
||||
})
|
||||
|
||||
flags = []
|
||||
if runway_months < threshold_months:
|
||||
flags.append(f"RUNWAY BELOW THRESHOLD: {runway_months:.1f} months < {threshold_months} month threshold.")
|
||||
if any(m["verdict"] == "AT-RISK" for m in milestones_out):
|
||||
flags.append("At least one value-inflection milestone is AT-RISK on current burn.")
|
||||
if trailing_burn > avg_burn * 1.15:
|
||||
flags.append(f"Burn accelerating: trailing burn ${trailing_burn:,.0f} > 115% of average ${avg_burn:,.0f}.")
|
||||
|
||||
return {
|
||||
"program": data.get("program", "UNSPECIFIED"),
|
||||
"cash_on_hand": cash,
|
||||
"average_burn_per_period": round(avg_burn, 2),
|
||||
"trailing_burn_per_period": round(trailing_burn, 2),
|
||||
"forward_run_rate_used": round(run_rate, 2),
|
||||
"runway_periods": round(runway_periods, 2),
|
||||
"runway_months_approx": round(runway_months, 1),
|
||||
"milestones": milestones_out,
|
||||
"flags": flags,
|
||||
"assumptions": [
|
||||
f"Forward run-rate = trailing {trailing_n}-period burn (recent-weighted, not lifetime average).",
|
||||
f"Period label '{label}' => {months_per_period} month(s) per period.",
|
||||
"Runway assumes flat forward burn; a funded ramp or hiring plan changes this.",
|
||||
"Milestone cash needs are cumulative-from-now as supplied; verify against the program budget.",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _render_human(r: dict) -> str:
|
||||
lines = [f"Burn & Runway: {r['program']}", "",
|
||||
f"Cash on hand: ${r['cash_on_hand']:,.0f}",
|
||||
f"Average burn/period: ${r['average_burn_per_period']:,.0f}",
|
||||
f"Trailing burn/period: ${r['trailing_burn_per_period']:,.0f}",
|
||||
f"Forward run-rate used: ${r['forward_run_rate_used']:,.0f}",
|
||||
f"Runway: {r['runway_periods']} periods (~{r['runway_months_approx']} months)",
|
||||
""]
|
||||
lines.append("Milestones:")
|
||||
for m in r["milestones"]:
|
||||
lines.append(f" [{m['verdict']}] {m['name']} (+{m['period_from_now']:.0f} periods, "
|
||||
f"needs ${m['cumulative_cash_needed']:,.0f})")
|
||||
lines.append("")
|
||||
if r["flags"]:
|
||||
lines.append("Flags:")
|
||||
for f in r["flags"]:
|
||||
lines.append(f" ! {f}")
|
||||
lines.append("")
|
||||
lines.append("Assumptions:")
|
||||
for a in r["assumptions"]:
|
||||
lines.append(f" - {a}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description="Compute R&D program burn, runway, and milestone alignment.")
|
||||
p.add_argument("--input", help="Path to JSON ledger")
|
||||
p.add_argument("--threshold-months", type=float, default=None, help="runway alert threshold (months)")
|
||||
p.add_argument("--output", choices=["human", "json"], default="human")
|
||||
p.add_argument("--sample", action="store_true", help="use the embedded sample")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
conf = _cfg.load_config() if _cfg else {}
|
||||
threshold = args.threshold_months if args.threshold_months is not None \
|
||||
else float(conf.get("runway_threshold_months", 6.0))
|
||||
data = SAMPLE if (args.sample or not args.input) else json.load(open(args.input))
|
||||
try:
|
||||
result = analyze(data, threshold)
|
||||
except ValueError as e:
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
if args.output == "json":
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(_render_human(result))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,189 @@
|
||||
#!/usr/bin/env python3
|
||||
"""capex_vs_opex_router.py - Decision-SUPPORT for R&D capitalize-vs-expense treatment.
|
||||
|
||||
Stdlib-only. Deterministic. NO LLM calls. This tool NEVER books an entry and NEVER
|
||||
auto-decides accounting treatment. It scores each cost item against capitalization
|
||||
criteria and ROUTES it to a named finance owner for the actual determination.
|
||||
|
||||
Criteria reflect IAS 38 (development-phase capitalization test) and US GAAP ASC 730
|
||||
(R&D expensed as incurred) / ASC 985-20 (internal-use & sold software). The six IAS 38
|
||||
development-phase conditions:
|
||||
1. technical feasibility established
|
||||
2. intention to complete
|
||||
3. ability to use or sell
|
||||
4. probable future economic benefit
|
||||
5. adequate resources to complete
|
||||
6. reliable measurement of expenditure
|
||||
|
||||
Verdicts:
|
||||
- CAPITALIZE-CANDIDATE (development phase, all criteria met) -> still routes to finance owner
|
||||
- EXPENSE (research phase, or criteria not met)
|
||||
- FINANCE-OWNER-REVIEW (ambiguous / partial criteria)
|
||||
|
||||
Usage:
|
||||
python3 capex_vs_opex_router.py --sample
|
||||
python3 capex_vs_opex_router.py --input costs.json --standard ifrs
|
||||
python3 capex_vs_opex_router.py --input costs.json --standard usgaap --output json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
import config_loader as _cfg
|
||||
except ImportError: # pragma: no cover
|
||||
_cfg = None
|
||||
|
||||
IAS38_CRITERIA = [
|
||||
"technical_feasibility",
|
||||
"intention_to_complete",
|
||||
"ability_to_use_or_sell",
|
||||
"probable_future_benefit",
|
||||
"adequate_resources",
|
||||
"reliable_measurement",
|
||||
]
|
||||
|
||||
# Profiles only annotate context; they do not change the accounting test.
|
||||
PROFILES = {
|
||||
"pharma-rd": "Most drug R&D is expensed; capitalization rare pre-approval.",
|
||||
"biotech": "Similar to pharma; pre-approval development typically expensed.",
|
||||
"medtech": "Some development capitalizable post-feasibility under IFRS.",
|
||||
"deep-tech": "Prototype-to-product transition is the key feasibility line.",
|
||||
"software-rd": "ASC 985-20 / IAS 38: capitalize after technological feasibility / working model.",
|
||||
"university-lab": "Grant-funded research almost always expensed per funder terms.",
|
||||
}
|
||||
|
||||
SAMPLE = {
|
||||
"standard": "ifrs",
|
||||
"items": [
|
||||
{
|
||||
"name": "Exploratory target screening",
|
||||
"phase": "research",
|
||||
"criteria": {},
|
||||
},
|
||||
{
|
||||
"name": "Pilot-line tooling for validated design",
|
||||
"phase": "development",
|
||||
"criteria": {
|
||||
"technical_feasibility": True, "intention_to_complete": True,
|
||||
"ability_to_use_or_sell": True, "probable_future_benefit": True,
|
||||
"adequate_resources": True, "reliable_measurement": True,
|
||||
},
|
||||
},
|
||||
{
|
||||
"name": "Software build (post working-model, pre-release)",
|
||||
"phase": "development",
|
||||
"criteria": {
|
||||
"technical_feasibility": True, "intention_to_complete": True,
|
||||
"ability_to_use_or_sell": True, "probable_future_benefit": True,
|
||||
"adequate_resources": False, "reliable_measurement": True,
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def route_item(item: dict, standard: str) -> dict:
|
||||
phase = (item.get("phase") or "").lower()
|
||||
crit = item.get("criteria", {}) or {}
|
||||
met = [c for c in IAS38_CRITERIA if crit.get(c)]
|
||||
missing = [c for c in IAS38_CRITERIA if not crit.get(c)]
|
||||
|
||||
# US GAAP ASC 730: R&D expensed as incurred (software is the main exception via ASC 985-20).
|
||||
if standard == "usgaap" and phase != "software-development":
|
||||
verdict = "EXPENSE"
|
||||
rationale = "ASC 730: R&D is expensed as incurred (non-software). Confirm software exceptions separately."
|
||||
owner = "R&D Finance Controller"
|
||||
elif phase == "research":
|
||||
verdict = "EXPENSE"
|
||||
rationale = "Research phase: cannot capitalize (IAS 38.54)."
|
||||
owner = "R&D Finance Controller"
|
||||
elif phase in ("development", "software-development") and not missing:
|
||||
verdict = "CAPITALIZE-CANDIDATE"
|
||||
rationale = "Development phase with all 6 IAS 38 criteria asserted. Routed for finance confirmation."
|
||||
owner = "R&D Finance Controller + External Auditor sign-off"
|
||||
else:
|
||||
verdict = "FINANCE-OWNER-REVIEW"
|
||||
rationale = f"Development phase but {len(missing)} criteria unmet/unstated: {', '.join(missing) or 'n/a'}."
|
||||
owner = "R&D Finance Controller"
|
||||
|
||||
return {
|
||||
"name": item.get("name", "UNNAMED"),
|
||||
"phase": phase or "UNSPECIFIED",
|
||||
"criteria_met": met,
|
||||
"criteria_missing": missing,
|
||||
"verdict": verdict,
|
||||
"rationale": rationale,
|
||||
"named_owner": owner,
|
||||
}
|
||||
|
||||
|
||||
def route(data: dict, standard: str, profile: str) -> dict:
|
||||
if profile not in PROFILES:
|
||||
raise ValueError(f"Unknown profile '{profile}'. Choose from {list(PROFILES)}.")
|
||||
items = [route_item(i, standard) for i in data.get("items", [])]
|
||||
return {
|
||||
"standard": standard,
|
||||
"profile": profile,
|
||||
"profile_note": PROFILES[profile],
|
||||
"items": items,
|
||||
"disclaimer": "DECISION SUPPORT ONLY. This tool does not book entries or decide treatment. "
|
||||
"A named finance owner (and auditor where required) makes the determination.",
|
||||
}
|
||||
|
||||
|
||||
def _render_human(r: dict) -> str:
|
||||
lines = [f"Capitalize-vs-Expense routing (standard: {r['standard']}, profile: {r['profile']})",
|
||||
f" {r['profile_note']}", ""]
|
||||
for it in r["items"]:
|
||||
lines.append(f"[{it['verdict']}] {it['name']} (phase: {it['phase']})")
|
||||
lines.append(f" {it['rationale']}")
|
||||
if it["criteria_missing"]:
|
||||
lines.append(f" missing/unstated: {', '.join(it['criteria_missing'])}")
|
||||
lines.append(f" -> route to: {it['named_owner']}")
|
||||
lines.append("")
|
||||
lines.append(f"!! {r['disclaimer']}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description="Route R&D costs to capitalize/expense/review (DECISION SUPPORT ONLY).")
|
||||
p.add_argument("--input", help="Path to JSON with items[]")
|
||||
p.add_argument("--standard", default=None, choices=["ifrs", "usgaap"],
|
||||
help="overrides onboarding accounting_standard")
|
||||
p.add_argument("--profile", default=None, choices=list(PROFILES),
|
||||
help="overrides onboarding default_profile")
|
||||
p.add_argument("--output", choices=["human", "json"], default="human")
|
||||
p.add_argument("--sample", action="store_true", help="use the embedded sample")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
conf = _cfg.load_config() if _cfg else {}
|
||||
profile = args.profile or conf.get("default_profile", "biotech")
|
||||
cli_standard = args.standard or conf.get("accounting_standard", "ifrs")
|
||||
data = SAMPLE if (args.sample or not args.input) else json.load(open(args.input))
|
||||
standard = data.get("standard", cli_standard) if (args.sample or not args.input) else cli_standard
|
||||
try:
|
||||
result = route(data, standard, profile)
|
||||
except ValueError as e:
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
return 2
|
||||
finance_owner = conf.get("finance_owner")
|
||||
if finance_owner:
|
||||
for it in result["items"]:
|
||||
it["named_owner"] = it["named_owner"].replace(
|
||||
"R&D Finance Controller", f"R&D Finance Controller ({finance_owner})")
|
||||
|
||||
if args.output == "json":
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(_render_human(result))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,115 @@
|
||||
#!/usr/bin/env python3
|
||||
"""config_loader.py - Customization loader for the research-finance skill.
|
||||
|
||||
Stdlib-only. Importable from the skill's other scripts. Precedence (highest wins):
|
||||
1. Project config: <cwd>/.research-ops/research-finance.json
|
||||
2. Global config: ~/.config/research-ops/research-finance.json
|
||||
3. Built-in DEFAULTS
|
||||
|
||||
Onboarding answers (written by onboard.py) live in these files; every tool in this
|
||||
skill reads them so the user's customization applies automatically.
|
||||
Set RESEARCH_OPS_NO_CONFIG=1 to ignore saved config.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
SKILL = "research-finance"
|
||||
|
||||
GLOBAL_CONFIG_DIR = Path.home() / ".config" / "research-ops"
|
||||
GLOBAL_CONFIG_PATH = GLOBAL_CONFIG_DIR / f"{SKILL}.json"
|
||||
PROJECT_CONFIG_DIRNAME = ".research-ops"
|
||||
|
||||
DEFAULTS: dict[str, Any] = {
|
||||
"version": 1,
|
||||
"skill": SKILL,
|
||||
"default_profile": "biotech",
|
||||
"default_fa_rate": None, # None => use the profile's default F&A rate
|
||||
"runway_threshold_months": 6,
|
||||
"accounting_standard": "ifrs",
|
||||
"finance_owner": None,
|
||||
"setup_completed_at": None,
|
||||
}
|
||||
|
||||
|
||||
def project_config_path(cwd: Path | None = None) -> Path:
|
||||
cwd = cwd or Path.cwd()
|
||||
return cwd / PROJECT_CONFIG_DIRNAME / f"{SKILL}.json"
|
||||
|
||||
|
||||
def _read_json(path: Path) -> dict[str, Any] | None:
|
||||
try:
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
return data if isinstance(data, dict) else None
|
||||
except (FileNotFoundError, json.JSONDecodeError, OSError):
|
||||
return None
|
||||
|
||||
|
||||
def _deep_merge(base: dict[str, Any], override: dict[str, Any]) -> dict[str, Any]:
|
||||
out = dict(base)
|
||||
for k, v in override.items():
|
||||
if isinstance(v, dict) and isinstance(out.get(k), dict):
|
||||
out[k] = _deep_merge(out[k], v)
|
||||
else:
|
||||
out[k] = v
|
||||
return out
|
||||
|
||||
|
||||
def load_config(cwd: Path | None = None) -> dict[str, Any]:
|
||||
config = dict(DEFAULTS)
|
||||
if os.environ.get("RESEARCH_OPS_NO_CONFIG") == "1":
|
||||
return config
|
||||
global_cfg = _read_json(GLOBAL_CONFIG_PATH)
|
||||
if global_cfg:
|
||||
config = _deep_merge(config, global_cfg)
|
||||
project_cfg = _read_json(project_config_path(cwd))
|
||||
if project_cfg:
|
||||
config = _deep_merge(config, project_cfg)
|
||||
return config
|
||||
|
||||
|
||||
def setup_completed() -> bool:
|
||||
cfg = _read_json(GLOBAL_CONFIG_PATH) or _read_json(project_config_path())
|
||||
return bool(cfg and cfg.get("setup_completed_at"))
|
||||
|
||||
|
||||
def write_config(config: dict[str, Any], scope: str = "global", cwd: Path | None = None) -> Path:
|
||||
path = project_config_path(cwd) if scope == "project" else GLOBAL_CONFIG_PATH
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("w", encoding="utf-8") as f:
|
||||
json.dump(config, f, indent=2, sort_keys=True)
|
||||
return path
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description=f"Inspect {SKILL} customization config.")
|
||||
p.add_argument("--show", action="store_true", help="Print the effective config")
|
||||
p.add_argument("--status", action="store_true", help="Print setup status + paths")
|
||||
p.add_argument("--sample", action="store_true", help="Print the built-in defaults")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
if args.sample:
|
||||
print(json.dumps(DEFAULTS, indent=2, sort_keys=True))
|
||||
elif args.status:
|
||||
print(json.dumps({
|
||||
"skill": SKILL,
|
||||
"global_config_path": str(GLOBAL_CONFIG_PATH),
|
||||
"global_config_exists": GLOBAL_CONFIG_PATH.exists(),
|
||||
"project_config_path": str(project_config_path()),
|
||||
"project_config_exists": project_config_path().exists(),
|
||||
"setup_completed": setup_completed(),
|
||||
}, indent=2))
|
||||
else:
|
||||
print(json.dumps(load_config(), indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,120 @@
|
||||
#!/usr/bin/env python3
|
||||
"""onboard.py - Onboarding questionnaire for the research-finance skill.
|
||||
|
||||
Stdlib-only. Asks the user a short set of questions BEFORE they build an R&D program
|
||||
budget, then writes the answers to a customization config read by every tool in this
|
||||
skill via config_loader.py. The answers become defaults for profile, F&A rate, runway
|
||||
threshold, accounting standard, and the named finance owner printed on routing outputs.
|
||||
|
||||
Modes: --show | --defaults | --set key=value (repeatable) | --reset | --scope {global,project}
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as _dt
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import config_loader as cfg # noqa: E402
|
||||
|
||||
NUMERIC_KEYS = {"default_fa_rate", "runway_threshold_months"}
|
||||
|
||||
QUESTIONS = [
|
||||
("default_profile",
|
||||
"1. What R&D area is this program?",
|
||||
["pharma-rd", "biotech", "medtech", "deep-tech", "software-rd", "university-lab"], str),
|
||||
("default_fa_rate",
|
||||
"2. F&A / indirect rate as a fraction (e.g. 0.55), or blank to use the profile default?",
|
||||
None, float),
|
||||
("runway_threshold_months",
|
||||
"3. Runway alert threshold in months (warn below this)?",
|
||||
None, float),
|
||||
("accounting_standard",
|
||||
"4. Which accounting standard governs capitalize-vs-expense?",
|
||||
["ifrs", "usgaap"], str),
|
||||
("finance_owner",
|
||||
"5. Named finance/controller owner who signs accounting treatment?",
|
||||
None, str),
|
||||
]
|
||||
|
||||
|
||||
def _print_questions() -> None:
|
||||
print(f"Onboarding questions — {cfg.SKILL}:\n")
|
||||
for _k, prompt, choices, _c in QUESTIONS:
|
||||
line = f" {prompt}"
|
||||
if choices:
|
||||
line += f" [{' / '.join(choices)}]"
|
||||
print(line)
|
||||
|
||||
|
||||
def run_interactive(config: dict) -> dict:
|
||||
print(f"Onboarding — {cfg.SKILL}. Press Enter to keep the current/default value.\n")
|
||||
for key, prompt, choices, caster in QUESTIONS:
|
||||
suffix = f" [{'/'.join(choices)}]" if choices else ""
|
||||
cur = f" (current: {config.get(key)})" if config.get(key) is not None else ""
|
||||
raw = input(f"{prompt}{suffix}{cur}: ").strip()
|
||||
if not raw:
|
||||
continue
|
||||
try:
|
||||
config[key] = caster(raw)
|
||||
except ValueError:
|
||||
print(f" ! invalid value for {key}, keeping current")
|
||||
return config
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description=f"Onboarding for the {cfg.SKILL} skill.")
|
||||
p.add_argument("--show", action="store_true")
|
||||
p.add_argument("--defaults", action="store_true", help="write built-in defaults, no prompt")
|
||||
p.add_argument("--set", action="append", default=[], metavar="key=value")
|
||||
p.add_argument("--reset", action="store_true")
|
||||
p.add_argument("--scope", choices=["global", "project"], default="global")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
if args.show:
|
||||
_print_questions()
|
||||
print("\nCurrent effective config:")
|
||||
print(json.dumps(cfg.load_config(), indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
if args.reset:
|
||||
path = cfg.project_config_path() if args.scope == "project" else cfg.GLOBAL_CONFIG_PATH
|
||||
if path.exists():
|
||||
path.unlink(); print(f"removed {path}")
|
||||
else:
|
||||
print(f"no config at {path}")
|
||||
return 0
|
||||
|
||||
config = cfg.load_config()
|
||||
|
||||
if args.set:
|
||||
for item in args.set:
|
||||
if "=" not in item:
|
||||
print(f"error: --set expects key=value, got '{item}'", file=sys.stderr)
|
||||
return 2
|
||||
k, v = item.split("=", 1)
|
||||
if k in NUMERIC_KEYS:
|
||||
try:
|
||||
v = float(v)
|
||||
except ValueError:
|
||||
pass
|
||||
config[k] = v
|
||||
elif not args.defaults:
|
||||
if sys.stdin.isatty():
|
||||
config = run_interactive(config)
|
||||
else:
|
||||
print("non-interactive shell: use --defaults or --set key=value. Showing questions:\n")
|
||||
_print_questions()
|
||||
return 0
|
||||
|
||||
config["setup_completed_at"] = _dt.datetime.now(_dt.timezone.utc).isoformat()
|
||||
path = cfg.write_config(config, scope=args.scope)
|
||||
print(f"saved {cfg.SKILL} customization -> {path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,165 @@
|
||||
#!/usr/bin/env python3
|
||||
"""program_budget_planner.py - Build a multi-period R&D program budget with F&A split.
|
||||
|
||||
Stdlib-only. Deterministic. NO LLM calls. Every output surfaces an explicit assumptions
|
||||
block: budget math without disclosed assumptions is theatre.
|
||||
|
||||
Takes work-package line items, applies the F&A (indirect) rate to the F&A-eligible base
|
||||
(MTDC-style: excludes capital equipment and the portion of subawards over $25k), computes
|
||||
fully-loaded cost, and rolls up per period.
|
||||
|
||||
Usage:
|
||||
python3 program_budget_planner.py --sample
|
||||
python3 program_budget_planner.py --input program.json --fa-rate 0.55 --periods 4
|
||||
python3 program_budget_planner.py --input program.json --profile biotech --output json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
try:
|
||||
import config_loader as _cfg
|
||||
except ImportError: # pragma: no cover
|
||||
_cfg = None
|
||||
|
||||
# Profile default F&A (indirect) rate and escalation assumption when not supplied in input.
|
||||
PROFILES = {
|
||||
"pharma-rd": {"default_fa_rate": 0.50, "annual_escalation": 0.03},
|
||||
"biotech": {"default_fa_rate": 0.55, "annual_escalation": 0.04},
|
||||
"medtech": {"default_fa_rate": 0.45, "annual_escalation": 0.03},
|
||||
"deep-tech": {"default_fa_rate": 0.40, "annual_escalation": 0.03},
|
||||
"software-rd": {"default_fa_rate": 0.30, "annual_escalation": 0.04},
|
||||
"university-lab": {"default_fa_rate": 0.585, "annual_escalation": 0.025},
|
||||
}
|
||||
|
||||
# Categories excluded from the F&A (MTDC) base.
|
||||
FA_EXEMPT_CATEGORIES = {"capital_equipment", "subaward_over_25k", "tuition", "patient_care"}
|
||||
|
||||
SAMPLE = {
|
||||
"program": "Next-Gen Assay Platform",
|
||||
"periods": 4,
|
||||
"work_packages": [
|
||||
{"name": "Personnel (FTEs)", "category": "personnel", "amounts": [320000, 330000, 340000, 350000]},
|
||||
{"name": "Consumables", "category": "supplies", "amounts": [60000, 65000, 70000, 70000]},
|
||||
{"name": "Sequencer", "category": "capital_equipment", "amounts": [180000, 0, 0, 0]},
|
||||
{"name": "CRO subaward", "category": "subaward_over_25k", "amounts": [100000, 100000, 0, 0]},
|
||||
{"name": "Travel", "category": "travel", "amounts": [12000, 12000, 12000, 12000]},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _period_sum(amounts: list, n: int, idx: int) -> float:
|
||||
return float(amounts[idx]) if idx < len(amounts) else 0.0
|
||||
|
||||
|
||||
def plan_budget(data: dict, fa_rate: float, periods: int) -> dict:
|
||||
wps = data.get("work_packages", [])
|
||||
direct_by_period = [0.0] * periods
|
||||
fa_base_by_period = [0.0] * periods
|
||||
line_items = []
|
||||
|
||||
for wp in wps:
|
||||
cat = wp.get("category", "other")
|
||||
amounts = wp.get("amounts", [])
|
||||
fa_eligible = cat not in FA_EXEMPT_CATEGORIES
|
||||
wp_total = 0.0
|
||||
for i in range(periods):
|
||||
amt = _period_sum(amounts, periods, i)
|
||||
direct_by_period[i] += amt
|
||||
if fa_eligible:
|
||||
fa_base_by_period[i] += amt
|
||||
wp_total += amt
|
||||
line_items.append({
|
||||
"name": wp.get("name", "UNNAMED"),
|
||||
"category": cat,
|
||||
"fa_eligible": fa_eligible,
|
||||
"total_direct": round(wp_total, 2),
|
||||
})
|
||||
|
||||
fa_by_period = [round(b * fa_rate, 2) for b in fa_base_by_period]
|
||||
loaded_by_period = [round(direct_by_period[i] + fa_by_period[i], 2) for i in range(periods)]
|
||||
|
||||
return {
|
||||
"program": data.get("program", "UNSPECIFIED"),
|
||||
"periods": periods,
|
||||
"fa_rate_applied": fa_rate,
|
||||
"line_items": line_items,
|
||||
"direct_by_period": [round(x, 2) for x in direct_by_period],
|
||||
"fa_base_by_period": [round(x, 2) for x in fa_base_by_period],
|
||||
"fa_by_period": fa_by_period,
|
||||
"fully_loaded_by_period": loaded_by_period,
|
||||
"total_direct": round(sum(direct_by_period), 2),
|
||||
"total_fa": round(sum(fa_by_period), 2),
|
||||
"total_fully_loaded": round(sum(loaded_by_period), 2),
|
||||
"assumptions": [
|
||||
f"F&A (indirect) rate applied: {fa_rate:.1%}. Confirm this is your negotiated NICRA, not an assumption.",
|
||||
f"F&A base excludes: {', '.join(sorted(FA_EXEMPT_CATEGORIES))} (MTDC-style base).",
|
||||
"Amounts are taken as-entered per period; no escalation applied unless baked into inputs.",
|
||||
"This is a planning estimate; a finance owner/controller validates the rate basis and booking.",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _render_human(r: dict) -> str:
|
||||
lines = [f"R&D Program Budget: {r['program']} ({r['periods']} periods)",
|
||||
f"F&A rate applied: {r['fa_rate_applied']:.1%}", ""]
|
||||
lines.append("Line items:")
|
||||
for li in r["line_items"]:
|
||||
tag = "F&A-eligible" if li["fa_eligible"] else "F&A-EXEMPT"
|
||||
lines.append(f" {li['name']:24s} {li['category']:20s} {tag:12s} ${li['total_direct']:,.0f}")
|
||||
lines.append("")
|
||||
hdr = " " + "".join(f"P{i+1:>14}" for i in range(r["periods"]))
|
||||
lines.append("Per-period rollup:" )
|
||||
lines.append(hdr)
|
||||
lines.append(" direct " + "".join(f"{v:>15,.0f}" for v in r["direct_by_period"]))
|
||||
lines.append(" F&A " + "".join(f"{v:>15,.0f}" for v in r["fa_by_period"]))
|
||||
lines.append(" loaded " + "".join(f"{v:>15,.0f}" for v in r["fully_loaded_by_period"]))
|
||||
lines.append("")
|
||||
lines.append(f"Total direct: ${r['total_direct']:,.0f}")
|
||||
lines.append(f"Total F&A: ${r['total_fa']:,.0f}")
|
||||
lines.append(f"Total fully-loaded: ${r['total_fully_loaded']:,.0f}")
|
||||
lines.append("")
|
||||
lines.append("Assumptions (state these alongside the number):")
|
||||
for a in r["assumptions"]:
|
||||
lines.append(f" - {a}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description="Build a multi-period R&D program budget with F&A split.")
|
||||
p.add_argument("--input", help="Path to JSON program with work_packages[]")
|
||||
p.add_argument("--profile", default=None, choices=list(PROFILES),
|
||||
help="overrides onboarding default_profile")
|
||||
p.add_argument("--fa-rate", type=float, default=None, help="Override F&A rate (fraction, e.g. 0.55)")
|
||||
p.add_argument("--periods", type=int, default=None, help="Number of periods")
|
||||
p.add_argument("--output", choices=["human", "json"], default="human")
|
||||
p.add_argument("--sample", action="store_true", help="use the embedded sample")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
conf = _cfg.load_config() if _cfg else {}
|
||||
profile = args.profile or conf.get("default_profile", "biotech")
|
||||
data = SAMPLE if (args.sample or not args.input) else json.load(open(args.input))
|
||||
periods = args.periods or int(data.get("periods", 4))
|
||||
# F&A precedence: CLI flag > onboarding default_fa_rate (if set) > profile default
|
||||
if args.fa_rate is not None:
|
||||
fa_rate = args.fa_rate
|
||||
elif conf.get("default_fa_rate") is not None:
|
||||
fa_rate = float(conf["default_fa_rate"])
|
||||
else:
|
||||
fa_rate = PROFILES[profile]["default_fa_rate"]
|
||||
|
||||
result = plan_budget(data, fa_rate, periods)
|
||||
if args.output == "json":
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
print(_render_human(result))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,156 @@
|
||||
---
|
||||
name: research-ops-skills
|
||||
description: Use when planning, funding, scoping, or synthesizing enterprise research across workstreams — clinical study design, R&D program finance, market sizing/surveys, or product/user research. Triggers on "design this clinical study", "what sample size", "R&D budget", "burn rate", "capitalize or expense", "TAM SAM SOM", "market sizing", "survey design", "segment the market", "plan user interviews", "usability test", "synthesize research insights". Forks context to route to one of four Research-Operations sub-skills (clinical-research, research-finance, market-research, product-research) and returns a digest. Distinct from ra-qm-team (regulatory submission), finance (corporate close/valuation), research/grants (funding discovery), product-team (persona/journey/live experiments), and marketing-skill (campaign analytics).
|
||||
context: fork
|
||||
version: 2.9.0
|
||||
author: claude-code-skills
|
||||
license: MIT
|
||||
tags: [research-ops, clinical-research, research-finance, market-research, product-research, rd, orchestrator]
|
||||
compatible_tools: [claude-code, codex-cli, cursor, antigravity, opencode, gemini-cli]
|
||||
---
|
||||
|
||||
# Research Operations — Domain Orchestrator
|
||||
|
||||
The Research Operations surface is **how the enterprise plans, funds, scopes, and synthesizes research** across four workstreams: clinical R&D, R&D finance, market research, and product research. This orchestrator forks its context, routes your inquiry to one of four sub-skills, then returns a digest. Heavy intake (protocol drafts, program ledgers, survey exports, interview transcripts) stays in the forked context.
|
||||
|
||||
This is the enterprise counterpart to the academic `research/` domain. If your question is about **finding** literature, grants, or patents, use `research/`. If it is about **planning, funding, scoping, or synthesizing** research as an operational discipline, you are in the right place.
|
||||
|
||||
## When to invoke
|
||||
|
||||
| Symptom | Sub-skill |
|
||||
|---|---|
|
||||
| "We're designing a Phase 2 trial — what's the endpoint and sample size?" | `clinical-research` |
|
||||
| "What's our R&D program burn, and is this cost CapEx or OpEx?" | `research-finance` |
|
||||
| "What's the TAM for this product, and how do we survey the segment?" | `market-research` |
|
||||
| "How many users do we interview, and how do we synthesize the findings?" | `product-research` |
|
||||
|
||||
## Routing logic (deterministic)
|
||||
|
||||
Same two-signal threshold pattern as `commercial-skills`. Single-signal → clarifying question. Mixed signals → highest-confidence first, chain second in a follow-up turn. Never silently chain.
|
||||
|
||||
### Signal table
|
||||
|
||||
| Signal class | Keywords | Sub-skill |
|
||||
|---|---|---|
|
||||
| **CLINICAL** | clinical trial, study design, protocol, endpoint, sample size, power, phase 1/2/3, biostatistics, eligibility, feasibility, estimand | `clinical-research` |
|
||||
| **RD_FINANCE** | R&D budget, program budget, burn, runway, F&A, indirect rate, overhead, capitalize vs expense, R&D capex, portfolio ROI, rNPV | `research-finance` |
|
||||
| **MARKET** | TAM, SAM, SOM, market sizing, survey design, sampling, margin of error, segmentation, competitive intelligence, market research | `market-research` |
|
||||
| **PRODUCT** | user interview, JTBD, usability test, concept test, prototype test, discovery research, research repository, insight synthesis, saturation | `product-research` |
|
||||
|
||||
## Workflow (Matt Pocock grill discipline)
|
||||
|
||||
Derived from Matt Pocock's `grill-with-docs` pattern: **explore-then-ask, one question per turn with a recommended answer, walk the decision tree depth-first, track dependencies, anchor every challenge in the research canon** (`references/` of each sub-skill).
|
||||
|
||||
### Step 1 — Explore before asking
|
||||
|
||||
Check the user's working directory first:
|
||||
- Is there a protocol draft, program ledger, TAM model, or interview guide already in the workspace?
|
||||
- Does the inquiry already disambiguate the lane (e.g., "what sample size for a two-arm trial" — that's `clinical-research`, no question needed)?
|
||||
- Is there an artifact filename that resolves the lane (`protocol.json` → clinical; `program-budget.json` → finance; `tam-model.json` → market; `interview-guide.md` → product)?
|
||||
|
||||
If the workspace resolves the lane, **route silently**.
|
||||
|
||||
### Step 2 — If still ambiguous, ONE forcing question with a recommended answer
|
||||
|
||||
Matt's rule: never bundle. Always recommend.
|
||||
|
||||
Pattern:
|
||||
```
|
||||
Q1/1: [precise question naming the two candidate lanes]
|
||||
Recommended: [Lane X, because <signal-table rationale>]
|
||||
|
||||
(Confirm, or override?)
|
||||
```
|
||||
|
||||
### Step 3 — Decision-tree walk for multi-lane inquiries
|
||||
|
||||
If the inquiry legitimately crosses two lanes (e.g., "design this trial AND budget it" = CLINICAL + RD_FINANCE), walk depth-first:
|
||||
|
||||
1. Highest-confidence lane first → run sub-skill in forked context → digest
|
||||
2. Ask: "Now run [second lane]? Recommended: yes, because [dependency]."
|
||||
3. Confirm before chaining.
|
||||
|
||||
Never silently chain.
|
||||
|
||||
### Step 4 — Invoke sub-skill in forked context
|
||||
|
||||
Forward original prompt + structured inputs (protocol JSON, program ledger CSV, market model, observation export).
|
||||
|
||||
### Step 5 — Return digest with cited canon challenge
|
||||
|
||||
≤ 200 words: analyzed, top 3 findings (anchored to a canon citation), top 3 next actions (named human owner where applicable), artifact path, and **one grill challenge** for the user. Examples:
|
||||
|
||||
- "Your power calc assumes a 0.5 effect size with no published anchor. ICH E9 requires a justified, clinically meaningful difference. Where did 0.5 come from?"
|
||||
- "Your TAM is a single top-down number (1% of a $40B market). Bessemer market-sizing discipline requires a bottoms-up cross-check. What's units × price × adoption?"
|
||||
|
||||
## Forcing-question library (grill-with-docs pattern)
|
||||
|
||||
Grill the user on lane-defining decisions before invoking the sub-skill. One per turn, recommended answer, canon citation:
|
||||
|
||||
- **CLINICAL lane**: "Is your primary endpoint a clinical outcome or a surrogate — and if surrogate, is it validated for this indication? Recommended: clinical outcome unless the surrogate is on FDA's validated table. Canon: FDA Surrogate Endpoint Table; BEST glossary."
|
||||
- **RD_FINANCE lane**: "Is this spend in the research phase or the development phase, and can you evidence technical feasibility? Recommended: research = expense; development = capitalize-candidate only with feasibility evidence, routed to a named finance owner. Canon: IAS 38; ASC 730."
|
||||
- **MARKET lane**: "Is your TAM top-down or bottoms-up — and have you computed it both ways to triangulate? Recommended: both; reconcile the delta. Canon: Bessemer / a16z market-sizing; Fermi estimation."
|
||||
- **PRODUCT lane**: "Is this study generative (discover problems) or evaluative (test a solution)? Recommended: name it first; the method follows. Canon: Rohrer's landscape of UX research methods (NN/g)."
|
||||
|
||||
Never run a sub-skill until the lane-defining decision is locked.
|
||||
|
||||
## Onboarding-first (per sub-skill)
|
||||
|
||||
Before invoking a sub-skill for the first time in a workspace, point the user at that skill's onboarding questionnaire so the tools run pre-configured to their context:
|
||||
|
||||
```bash
|
||||
python3 skills/<sub-skill>/scripts/onboard.py # interactive Q&A
|
||||
python3 skills/<sub-skill>/scripts/onboard.py --show # questions + current config
|
||||
```
|
||||
|
||||
Each sub-skill has its **own** question set (clinical: area/alpha/power/dropout/owners · finance: area/F&A/runway/standard/owner · market: profile/confidence/MoE/method · product: profile/insight-threshold/method/stakes). Answers persist to `~/.config/research-ops/<sub-skill>.json` (or `./.research-ops/<sub-skill>.json` with `--scope project`) and are consumed automatically by every tool in that skill. Customization is mandatory discipline here, not decoration — surface the onboarding step when a user starts a fresh research workstream.
|
||||
|
||||
## Autoresearch handoff (isolated, opt-in)
|
||||
|
||||
Each sub-skill ships its own `skills/<sub-skill>/scripts/ar_evaluator.py` — an **isolated** bridge to `engineering/autoresearch-agent`. Invoke autoresearch **only when the user explicitly asks** to "optimize", "improve", or "run a loop". The handoff is per-skill (no shared coupling): the loop edits the skill's input file and the evaluator scores it (clinical → `feasibility_composite` higher; finance → `runway_months` higher; market → `tam_divergence` lower; product → `validated_insights` higher). Never auto-start a loop; never let the loop edit the evaluator.
|
||||
|
||||
## Assumptions
|
||||
|
||||
1. User has research authority OR is preparing analysis for someone who does.
|
||||
2. User wants **deterministic decision support**, not the final answer — a clinician approves the protocol, a controller books the entry, the human picks the market number.
|
||||
3. Inputs may be partial — every sub-skill ships a templated sample so the user can see the shape before filling in their own.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Not an EDC, clinical-trial-management system, accounting system, survey platform, or research repository.
|
||||
- Does not give clinical, accounting, or legal advice as fact. Every output is **a recommendation + named human owner**.
|
||||
- Does not store research history across sessions.
|
||||
|
||||
## Distinct from
|
||||
|
||||
- **`research/` (academic)** — that domain **finds** literature, grants, and patents. This domain **plans, funds, scopes, and synthesizes** research.
|
||||
- **`ra-qm-team`** — that's **regulatory/QM submission** (ISO 13485/14971, MDR, FDA 510(k)/PMA/QSR). clinical-research designs the **study**; it routes submission out to ra-qm-team.
|
||||
- **`finance/financial-analysis`** — that's **corporate close + valuation**. research-finance manages **R&D program/portfolio spend**.
|
||||
- **`research/grants`** — that's **funding discovery**. research-finance manages **money already won**.
|
||||
- **`product-team`** — that's **persona/journey artifacts, discovery sprints, and live A/B experiments**. product-research is the **method + repository discipline**.
|
||||
- **`marketing-skill`** — that's **campaign analytics and demand-gen**. market-research is **upstream methodology**.
|
||||
|
||||
## Output artifacts
|
||||
|
||||
| Sub-skill | Artifact |
|
||||
|---|---|
|
||||
| clinical-research | `protocol_synopsis.md` + `sample_size.json` |
|
||||
| research-finance | `rd_program_budget.md` + `capex_opex_routing.json` |
|
||||
| market-research | `market_sizing.md` + `sample_plan.json` |
|
||||
| product-research | `research_plan.md` + `insight_synthesis.json` |
|
||||
|
||||
## Anti-patterns (do not)
|
||||
|
||||
- ❌ Present a clinical power/endpoint output as fact — it is an **estimate** with a named clinical owner
|
||||
- ❌ Auto-decide capitalize-vs-expense — route to a **named finance owner**
|
||||
- ❌ Report a market size as a single unsourced number — show **method + both-ways triangulation + assumptions**
|
||||
- ❌ Assert a product insight from a single participant — flag it as an **anecdote**
|
||||
- ❌ Run all 4 sub-skills "to be thorough" — pick one, digest, chain if needed
|
||||
|
||||
## References
|
||||
|
||||
- Clinical canon: ICH E8(R1)/E9/E9(R1), CONSORT, SPIRIT, FDA Multiple Endpoints
|
||||
- R&D finance canon: IAS 38, ASC 730, 2 CFR 200, Cooper stage-gate
|
||||
- Market canon: Cochran, Dillman, Kotler, Bessemer market-sizing
|
||||
- Product canon: Nielsen, Guest et al., Christensen JTBD, ResearchOps/Polaris
|
||||
- Path-B build pattern: `documentation/implementation/research-ops-expansion-plan.md`
|
||||
Reference in New Issue
Block a user