pm-experimentation-ab

• SKILL.mdcore · 12,794 bytes
• SKILL.mdcore · 12,794 bytes

# A/B Testing & Experimentation Skill

## How this skill behaves (read first)

This is a **generative** skill (it designs or revises an experiment plan, and can review an existing test setup). Experimentation is where an AI assistant's defaults are quietly wrong in ways that produce confident, false conclusions. Claude will happily "set up an A/B test" that **declares a winner on a handful of users**, **peeks and stops the moment p < 0.05**, **optimizes one metric with no guardrail** for the damage it does elsewhere, runs for three weekdays, and ends at "B won" with no segments, no long-term read, and nothing written down. A bad experiment is worse than none — it launders a guess into "data."

So this skill gates, and the gate is statistical discipline:

1. **Establish context** — is a controlled experiment even the right tool here, what single decision it informs, and the risk level (these set confidence, split, and duration).
2. **Apply the always-true core** — pre-register the hypothesis and success criteria, size the test *before* running it, run a full cycle, use one primary metric + guardrails, interpret past "it won," and close the learning loop.
3. **Surface the context-dependent decisions** (confidence threshold, traffic split, A/B vs. multivariate, metric type, leading/long-term indicators, test prioritization) with trade-offs; let the user choose.

Then it **hands off to `pm-okr-metric-validity-audit`** (are the chosen metrics valid, not vanity?) and **`pm-assumption-rigor-audit`** (is the hypothesis the riskiest, falsifiable, pre-committed?).