Prompt pack3 steps5 prompts
Design & Analyze A/B Tests
Use this when you will ship two versions to traffic and let a metric decide. Hypothesis, design, sample size, then run, then decide. This is not a survey and not a usability test.
01Hypothesis and design
If we change X, metric Y moves because Z. One change. Primary metric, guardrails, how long, how many people. Write it down before anyone builds.
- MarketingWrite A/B test hypothesesCopy-paste experimentation prompt for clear hypotheses, metrics, and test designs.
- ResearchA/B Test Design TemplateDesign rigorous A/B tests with clear hypotheses, metrics, and success criteria.
- ResearchA/B Test Sample Size GuideCalculate required sample size and test duration for statistically valid experiments.
02Run
Build, QA tracking, launch. Watch sample ratio and guardrails. Do not peek at the primary metric to “see if we can stop.”
03Decide
Significance is not the same as a change worth shipping. Effect size, guardrails, then ship, iterate, or drop. Write it down either way.
Frequently asked questions
Can we stop when it looks significant?
No. Wait for the sample you planned, unless the product is on fire.
We do not have the traffic. Still run this?
No. Use interviews or a survey. An underpowered test will give you a number you cannot defend.
People cannot finish the task. A/B it?
No. Run usability first. Shipping two broken variants measures which broken one loses slower.