The number of people, sessions or conversions a test needs before its result can be trusted. Smaller changes and lower conversion rates need larger samples.
Formula
Rough approximation per version (80% power, 5% significance): n ≈ 16 × p × (1 − p) ÷ δ²
Example
Meera's page converts at 2% and she wants to detect a rise to 2.5%. The approximation gives 16 × 0.02 × 0.98 ÷ 0.005² = 12,544 sessions per version.
How you'll hear it
“At our traffic, the sample size means this test needs two months, so let's test a bigger change.”
Don't confuse it with
- Statistical significance: the judgement made once you have the sample.
Related terms
Want to use this at work? AI-led PPC Excellence is aFactor's 16-week program for paid ads.
Sources
- Sample Size Calculator, Evan Miller
- How Not To Run an A/B Test, Evan Miller, 2010