Answer · Conversion Engineering
How Long Should an A/B Test Run?
The short answer
Until it reaches the pre-calculated sample size from your sample-size calculator, AND a minimum of 14 days to cover one full business cycle (day-of-week variation). Stopping early because the variant “is winning” inflates false-positive rates 3-5x. Set the duration up front, then don’t peek.
№ 01The longer answer
The two duration rules: (1) reach the pre-calculated sample size, (2) run at least 14 days. The 14-day minimum exists because Tuesday users behave differently from Saturday users, and a 7-day test misses one of the two weekly cycles.
Calculate sample size BEFORE starting. Inputs: baseline conversion rate, MDE (minimum detectable effect), confidence (95%), power (80%). Use Optimizely’s calculator or Evan Miller’s tools. The output is sessions per variant required.
Peeking and stopping early inflates false positives. Frequentist A/B tests assume a fixed sample size; stopping when the variant looks good is statistical cheating. If you want to peek and stop early, use sequential (Bayesian) tests on purpose — tools like VWO SmartStats or GrowthBook handle this correctly.
If the test hasn’t reached significance after 4-6 weeks, the result is probably a null result (no real difference). Don’t extend indefinitely hoping for significance — document the null and move on. Null results are valid CRO output.
№ 02What if my test reaches significance in 5 days?
If you pre-calculated the sample size and you genuinely hit it in 5 days, you can stop — but verify you covered at least one weekly cycle. If you didn’t pre-calculate sample size, the early “win” is probably noise. Pre-calculation is mandatory.
№ 03Should I run tests over holidays?
No — major holidays (US: Thanksgiving week, Christmas-New Year) and Black Friday distort traffic and behavior. Pause running tests during these windows. Resume after seasonal patterns stabilize.
№ 04What if I want to peek?
Use Bayesian/sequential testing methods on purpose. Tools that support it: VWO SmartStats, GrowthBook, Optimizely Stats Engine. The math handles peeking correctly; frequentist math does not.
Go deeper
Related questions
Three Ways to Start · No Sales Pitch
Want this answered for your business?
$500 audit. 5-day delivery. Refundable on engagement.