Answer · Conversion Engineering
What’s an A/B Test?
The short answer
An A/B test (or split test) shows two versions of a page (control vs treatment) to randomly-assigned visitors at the same time, then compares conversion rates between the two groups. It requires enough traffic to detect a meaningful difference with statistical confidence — usually 5,000+ sessions per variant for a typical B2B baseline.
№ 01The longer answer
A/B testing is the gold standard for causal CRO inference. By randomizing assignment, the test rules out confounding variables (seasonality, traffic-source mix, day-of-week effects) that would contaminate a simple pre/post comparison.
The math: you need a pre-calculated sample size based on baseline conversion rate, minimum detectable effect (MDE), confidence level (typically 95%), and statistical power (typically 80%). For a 3% baseline and 20% MDE, that’s about 4,800 sessions per variant.
Most mid-market sites can’t complete A/B tests on primary conversions because traffic volume is too low. A site at 8K sessions/month would need 6+ weeks per test, long enough for seasonality to contaminate results. For these sites, sequential (Bayesian) tests, pre/post analysis with proper baselines, or microconversion-only tests are the workable alternatives.
A/B testing tools: Optimizely (enterprise, $2K+/mo), VWO ($199+/mo), Convert.com ($99+/mo), GrowthBook (open source, free), Google Optimize (discontinued September 2023 — Google doesn’t want you doing this anymore in their free tools).
№ 02What’s an A/B/n test?
A/B/n is a test with 3+ variants (A vs B vs C, etc). The math gets worse fast — you need more total traffic to reach significance across more variants, and you have to correct for multiple comparisons. Stick with 2-variant tests unless traffic is abundant.
№ 03Can I run multiple A/B tests at once?
Yes, if they’re on different pages or different segments. Running multiple tests on the same page creates interaction effects you can’t cleanly disentangle. Use a multivariate test (MVT) framework if you genuinely need to test multiple page elements at once.
№ 04What if my test isn’t reaching significance?
Either your sample size is too small (run longer or accept the null result), your MDE is unrealistic (your real lift is smaller than what you can detect at this volume), or there’s genuinely no difference (which is also a valid result — document it and move on).
Go deeper
Related questions
Three Ways to Start · No Sales Pitch
Want this answered for your business?
$500 audit. 5-day delivery. Refundable on engagement.