Individual skill
ab-test-analyzer
Analyzes an A/B experiment to a defensible verdict - sample-size and minimum-detectable-effect math, two-proportion significance testing with confidence intervals, and checks for the traps that fake a win (peeking, sample ratio mismatch, multiple comparisons). Use when someone asks "is this test significant", "did variant B win", "how long should I run this experiment", "what sample size do I need", or shows experiment results and asks whether to ship. Do NOT use for logging and organizing ML training runs - use experiment-tracking instead; for causal questions without a randomized experiment use causal-inference; for general funnel or metric investigation use funnel-analysis or product-analytics.
skills/ab-test-analyzer/SKILL.md
Source commit 97a13607d669157a4a16d5ba79b77f208319f7b9
What was checked
- Matches the observed repository HEAD
- Yes
- Scanner rules are current
- No
- Repository scan complete
- No
- Skill scan complete
- No
- Observed behavior
- Not checked yet
- Last checked
- Nov 6, 2026
These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.
Install this skill
Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.
An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.
Repository context
- Repository stars (not skill ratings)
- 24
- Repository last updated
- Oct 13, 2026
- License
- MIT
Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.
- claude Repository topic
- claude-skills Repository topic
- skill-me Repository topic
- testing Derived category
- data Derived category
Files and excerpt
File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.
Read excerpt
# A/B Test Analyzer
Read experiments like a skeptic: most "wins" are noise until proven otherwise. The costly mistake this skill prevents is shipping on a peeked, underpowered, or mis-split test - a false positive that ships is worse than no test, because the team now defends a change that does nothing. Every verdict here comes with the interval, the checks that passed, and the checks that did not.
## Operating procedure
Order is mandatory: validity checks (Step 3) come before the significance math (Step 4), because a significant result on an invalid test is still invalid.
### Step 1: Gather inputs
Collect before computing anything; label unknowns as guesses.
1. The single primary metric and its type (conversion rate, revenue per user, retention).
2. The hypothesis and the minimum effect that would matter to the business - the smallest lift worth the cost of shipping and maintaining the change. If the requester has no number, help set one; without it "significant" is uninterpretable.