A/B test significance calculator.

What statistical significance means

If version B converts at 3.4% and A at 3.0%, B might really be better, or you might just have had a lucky week. Significance tells you how unlikely a difference this big would be if the two versions were actually the same. At 95% confidence, there's less than a 5% chance you'd see a gap like this by luck alone.

This calculator uses a standard two-proportion z-test, the same method most A/B testing tools use for conversion rates.

Common A/B testing mistakes

• Stopping the test the moment it looks significant. Decide the sample size first and wait for it, or you'll often declare false winners.
• Testing with too little traffic. Small sites may need weeks to detect a modest improvement.
• Running a test for less than a full week, so weekday and weekend visitors aren't both counted.
• Testing tiny changes, like a button colour, that can't produce a difference big enough to measure.
• Changing several things at once, then not knowing which one made the difference.

Sample size: plan before you test

The sample size tab tells you how many visitors each version needs to reliably detect the improvement you care about. The smaller the improvement and the lower your conversion rate, the more traffic you need. If the number is more traffic than you get in a month, test a bolder change, or improve the page based on research instead. That's often the better route for smaller sites, and it's how our conversion rate optimisation work usually starts.