An experiment where you show two versions of something, such as a page, ad or email, to randomly split groups and compare which performs better on one measure.
Formula
Rough sample size per version (80% power, 5% significance): n ≈ 16 × p × (1 − p) ÷ δ², where p is the current conversion rate and δ is the change you want to detect. This is an approximation.
Example
Meera's product page converts at 2% and she wants to detect a rise to 2.5%. Using the approximation, she needs about 16 × 0.02 × 0.98 ÷ 0.005² = 12,544 sessions on each version, 25,088 in all.
How you'll hear it
“Version B is ahead after a week, but we're nowhere near sample size, so we keep it running.”
What it helps you decide
Which version to keep, based on evidence rather than opinion, and how long a test must run before you can trust it.
Where people misread it
People stop the test as soon as one version looks ahead. With small numbers, early leads are often chance, and checking every day and stopping at the first good result makes false winners much more likely.
The story
Meera tests a new product page. Version B adds size-fit photos on real customers. After one week, B looks like a clear winner.
| Version A: 30 orders from 1,500 sessions | 2.0% |
| Version B: 42 orders from 1,500 sessions | 2.8% |
A 40% improvement. She is ready to switch. But a standard two-proportion test on these numbers gives a p-value of about 0.15. A gap this size turns up by chance roughly one time in seven when there is no real difference.
Before starting, she should have worked out the sample she needs. Her page converts at 2%, and she wants to detect a rise to 2.5%.
| Current conversion rate, p | 2% |
| Change to detect, δ | 0.5 points |
| Sessions per version | 12,544 |
| Total sessions | 25,088 |
| Sessions per month on her product pages | 12,000 |
| Time to run | about 2.1 months |
|---|
One week gave her 3,000 sessions, about an eighth of what she needs. She leaves the test running until both versions reach the planned sample, then decides once.
All figures are illustrative. The 16 × p(1 − p) ÷ δ² rule is an approximation; use a proper calculator for real tests.
How many sessions your A/B test needs
The example's numbers are filled in. Replace them with yours.
Decide the sample size before you start, and do not stop early because one version looks ahead.
Don't confuse it with
- Lift test / holdout: compares ads against no ads; an A/B test compares two versions against each other.
- Multivariate test: tests several elements at once.
Related terms
Want to use this at work? AI-led PPC Excellence is aFactor's 16-week program for paid ads.
Sources
- A/B testing, Optimizely Optimization Glossary
- How Not To Run an A/B Test, Evan Miller, 2010
- Sample Size Calculator, Evan Miller