wehub-resource-sync
11 KiB
Experiment Loop with Pre-Registered Gates
name: Experiment Loop with Pre-Registered Gates
slug: experiment-loop-with-pre-registered-gates
tier: forward-deployed (operations)
role: fdm
status: template
score: 43 (demand 4, pain 4, differentiation 3, usability 4, connectors 5)
intent: Basano on the test design: the stopping rule, MDE, and sample-size math
pre-registered before any variant ships, the SEO safety of variants gated,
and the verdict computed only against the pre-registered rule
when to use: a mechanism hypothesis worth a controlled test on a property with enough
traffic to power it
when not to use: finding the hypothesis in the first place (Conversion-by-Source Diagnosis);
a defect fix rather than a test (the ordinary held-fix flow)
validation note: gated on adequate traffic on a designated property; it holds at template
until a designated property has the traffic and a mechanism to test
Connectors
connectors:
- capability: experiment.read
access: read
- capability: analytics.read
access: read
- capability: crawl.read
access: read # variant SEO-safety checks on the built variant
- capability: repo.change
access: write-held # the variant and its config land as held changes
The variant and its config land held; a human allocates traffic and launches. Traffic allocation is never an agent action. The verdict is computed against the pre-registered rule and nothing else.
Prerequisites
- Claude with the catalog installed:
/plugin marketplace add rampstackco/claude-skills - A hypothesis from Conversion-by-Source Diagnosis (mechanism, expected effect, segment), or an equivalent stated the same way.
- Enough traffic in the target segment to power a test; if the segment cannot reach the sample the math requires, the honest outcome is not to run.
- The experiment platform (experiment.read) and the analytics that will measure the primary metric.
- A person who allocates traffic and launches; the workflow never does.
Phases
Phase 1: Intake · lane: convergent (Tholo)
Skills: analytics-strategy Capability class: experiment.intake (substitute equivalents if off-catalog) Input: the hypothesis from Conversion-by-Source Diagnosis (mechanism, expected effect, segment) Run:
Invoke analytics-strategy to frame the incoming hypothesis as a testable spec:
the mechanism it claims (what about the page or offer would move the metric,
and why), the expected effect size, the primary metric it would move, and the
exact segment. A hypothesis that arrives as an aspiration rather than a
mechanism goes back to Conversion-by-Source Diagnosis; this workflow tests
mechanisms, not hopes. Produce the framed spec.
Output artifact: the framed experiment spec (mechanism, expected effect size, primary metric, segment) Done when: the hypothesis is framed as a mechanism with an expected effect size, a primary metric, and a segment, or returned to Conversion-by-Source Diagnosis as too vague to test Fails look like: testing an aspiration. "Improve the CTA" names no mechanism and predicts no effect, so whatever the test does, nothing is learned; the spec has to say why the gap exists before a test can falsify it.
Phase 2: Pre-registration · lane: gate (Basano)
Skills: experiment-design, experimentation-analytics Capability class: experiment.pre-registration (substitute equivalents if off-catalog) Input: the framed experiment spec; the segment's available traffic Run:
Invoke experiment-design and experimentation-analytics to pre-register the
test BEFORE anything ships: the stopping rule, the minimum detectable effect,
the sample-size math the MDE and the segment's traffic imply, the primary
metric, and the window. Write them down and freeze them. If the segment cannot
reach the required sample in a reasonable window, the pre-registration verdict
is "insufficient n, do not run", which is a respectable and final outcome, not
a delay. Report the pre-registration, or the not-run verdict, with the math.
Output artifact: the frozen pre-registration (stopping rule, MDE, sample size, primary metric, window), or the "insufficient n, do not run" verdict with its math Done when: the test is pre-registered with all five fields frozen, or it is declared underpowered and not run, with the sample-size math shown either way Fails look like: shipping first and pre-registering after. A stopping rule and a metric written once the variant is believed in are fitted to the hope, and the test becomes a search for a number that agrees with it.
Phase 3: Variant SEO-safety gate · lane: gate (Basano)
Skills: seo-technical Capability class: experiment.seo-safety (substitute equivalents if off-catalog) Input: the built variant (crawl.read); the pre-registration Run:
Invoke seo-technical against the built variant. Check that the variant is safe
to serve to search: no cloaking risk (the variant does not show crawlers and
users materially different content in a way search treats as deceptive),
canonical consistency across the variant and the control, and no layout-shift
or core-vitals regression the variant introduces. Report a verdict per check
with the evidence. Report only.
Output artifact: the SEO-safety report (cloaking, canonical consistency, layout-shift, each with a verdict) Done when: the variant carries a verdict on cloaking risk, canonical consistency, and layout-shift, all clear or the risk flagged Fails look like: an experiment that wins by getting deindexed. A variant that trims the content search ranked for can lift the on-page metric while the page quietly falls out of the results, and the test would call that a win.
Phase 4: Launch as a held config change · lane: divergent (human)
Skills: none; the launch and the traffic allocation are deliberately human Input: the pre-registration and the passing SEO-safety report Run: the variant and its experiment config land as a HELD change (repo.change). A person allocates traffic and launches the test; traffic allocation is never an agent action, because it is a live change to what real users see. The human launches against the frozen pre-registration, allocating exactly the design the math called for, and records the launch. Output artifact: the launched experiment, recorded against its pre-registration, with the variant and config as held changes a human merged Done when: the experiment is launched by a human at the pre-registered allocation, with the config held and merged by a person Fails look like: an agent turning traffic on. Allocating traffic is a production act on real users, and it stays with a person the same way merging and deploying do.
Phase 5: The verdict · lane: gate (Basano)
Skills: experimentation-analytics Capability class: experiment.verdict (substitute equivalents if off-catalog) Input: the running experiment's results; the frozen pre-registration Run:
Invoke experimentation-analytics to compute the verdict ONLY against the
pre-registered rule, at the pre-registered point. The stopping rule decides
when the test ends; interim looks are not stopping decisions, and a result
that looks significant early is not a verdict until the pre-registered point.
The primary metric is the one pre-registered, not whichever moved. Report the
verdict (win, loss, or inconclusive) against the pre-registered rule, with the
numbers. Report only.
Output artifact: the verdict against the pre-registered rule (win, loss, or inconclusive) with the numbers at the pre-registered point Done when: the verdict is computed at the pre-registered point against the pre-registered primary metric and stopping rule, with the numbers attached Fails look like: peeking, then stopping on a good look. Checking the test daily and stopping the day it crosses significance turns noise into a false win, because the more often you look, the more often chance alone crosses the line.
Phase 6: Closeout · lane: convergent (Tholo)
Skills: experimentation-analytics Capability class: experiment.closeout (substitute equivalents if off-catalog) Input: the verdict; the original hypothesis and its Conversion-by-Source finding Run:
Invoke experimentation-analytics to close the loop. Route the verdict and the
hypothesis outcome back to Conversion-by-Source Diagnosis as calibration: a
hypothesis that won, lost, or came back inconclusive updates that workflow's
hit-rate, which is what makes its next ranking better. If a winning variant is
to consolidate into the live page, it re-enters the SEO-safety check on the
consolidated version before it ships through the ordinary held-fix flow; the
experiment never publishes itself. Record the closeout.
Output artifact: the routed verdict and hypothesis outcome to Conversion-by-Source Diagnosis, and the consolidation path (with its SEO-safety re-check) recorded where a variant won Done when (public gate): the verdict and hypothesis outcome are recorded and routed to Conversion-by-Source Diagnosis, and any winning variant is queued for the ordinary held-fix flow with an SEO-safety re-check, not published from here Operated-layer note: in an operated deployment the pre-registration, the verdict, and their agreement land as an agreement-log row; the verdict-versus-pre-registration agreement is what calibrates this lane (AGREEMENT-LOG.md) Fails look like: shipping the winner without re-checking the consolidated version. A variant proven safe in isolation can regress SEO once it is merged into the real page with the rest of the layout, and the win does not carry a waiver for that.
Failure modes
- Peeking (Phase 5's failure): interim looks becoming stopping decisions, which manufactures false wins.
- Post-hoc metric switching: reporting whichever metric moved instead of the pre-registered primary.
- Retrofitted pre-registration (Phase 2's failure): registering the rule after the variant is believed in.
- Shipping the winner without the SEO-safety re-check on the consolidated version (Phase 6's failure).
- Underpowered tests run anyway: ignoring the "insufficient n, do not run" verdict and running a test that cannot detect the effect; not-rankable is a verdict here too.
- An agent allocating traffic (Phase 4's failure): a production act on real users that stays human.
Worked example
Pending. Populates when this workflow is executed as written on a designated property with adequate traffic to power a test; it holds at template until a designated property has both the traffic and a mechanism hypothesis to run. Status flips to validated when a run record links here.
Boundaries
- Conversion-by-Source Diagnosis supplies this workflow's intake (its ranked hypotheses) and consumes its verdicts as calibration; the two form the diagnose-then-test loop, and neither runs the other's step.
- Revenue Tracking Integrity underwrites the measurement this workflow trusts: a verdict computed over a money path that does not survive redirects or consent is a verdict over noise.
- Content Pipeline with Prove Gates owns the publish path; a winning variant consolidates through the ordinary held-fix flow with an SEO-safety re-check, and the experiment never self-selects into publishing.