What is statistical significance in A/B testing?

Statistical significance tells you whether the difference between your control and variant conversion rates is real or just due to random chance. A result is typically considered significant at 95% confidence, meaning there is less than a 5% probability the difference occurred by chance.

How many visitors do I need for a valid A/B test?

The sample size depends on your baseline conversion rate and the minimum detectable effect you want to measure. Generally, you need at least 1,000 visitors per variation for small differences (1-2% lift) and 300-500 per variation for larger differences (5%+ lift). Lower baseline conversion rates require larger samples.

What p-value should I use for my A/B test?

Most A/B tests use a p-value threshold of 0.05 (95% confidence). For high-stakes decisions like pricing changes, use 0.01 (99% confidence). For low-risk UI tweaks, 0.10 (90% confidence) may be acceptable. The lower the p-value, the stronger the evidence that the observed difference is real.