A way of judging whether a difference in a test is likely to be real rather than chance. A result is usually called significant when the chance of seeing that difference with no real effect is below 5%.
Example
After a week, Meera's page B converts at 2.8% (42 orders from 1,500 sessions) and page A at 2.0% (30 from 1,500). A two-proportion test gives a p-value of about 0.15, so the difference is not significant yet.
How you'll hear it
“B is ahead but not significant, so we can't call it.”
Don't confuse it with
- Practical significance: whether the difference is big enough to matter in rupees.
- Sample size: how much data you need before significance means anything.
Related terms
Want to use this at work? AI-led PPC Excellence is aFactor's 16-week program for paid ads.
Sources
- Statistical significance, Optimizely Optimization Glossary
- How Not To Run an A/B Test, Evan Miller, 2010