A/B Test Sample Size Calculator

MDE is the smallest relative lift that would actually change your decision. A common starting point is 10 to 20 percent relative for a homepage or pricing page test where a real winner moves a top-line metric. For deeper funnel steps (checkout, onboarding) lifts can be larger, so MDE of 15 to 30 percent is reasonable. The temptation is to set MDE very low to feel sensitive, but smaller MDEs require dramatically larger samples. Pick the MDE that matches the business decision, not the curiosity.

Two reasons. First, conversion rates are noisy, so distinguishing a true 5 percent lift from random variation in a 3 percent baseline conversion takes tens of thousands of users per variant. Second, the math compounds: lower baseline rates, smaller MDEs, higher confidence, and higher power all push the number up nonlinearly. If your sample size looks impossibly large, the realistic move is to raise your MDE, run the test only on a high-traffic page, or test a higher-baseline metric upstream.

Default to two-tailed. A two-tailed test detects changes in either direction, which is honest because most product changes can also hurt. One-tailed tests halve the sample size but only catch lifts in the predicted direction, and if your change quietly drops conversion you will not notice. Use one-tailed only when a regression is impossible by construction (rare) or when you have explicit downstream guardrail metrics catching regressions.

Industry standard is 95 percent significance (alpha = 0.05) and 80 percent power (beta = 0.20). These are not laws of physics, but they balance false positives and false negatives reasonably for product experiments. Push significance to 99 percent only for high-stakes pricing changes or anything that touches revenue directly. Push power to 90 percent if missing a real effect is more costly than a false alarm. Anything below 80 percent power means you are likely running tests that cannot conclude.

You inflate your false positive rate. The standard test math assumes one look at the data after the planned sample size is reached. Every time you peek and consider stopping early, you re-roll the dice on a false win. Teams that peek and call winners early end up shipping a parade of fake wins that quietly underperform. If you genuinely need to monitor a test in flight, use sequential testing or always-valid p-values, which are designed for repeated peeking.

At minimum one full business week, ideally two, regardless of how fast you hit the calculated sample size. Conversion patterns shift across weekdays and weekend, and a test that finished in 36 hours during a Tuesday traffic spike has not seen Friday or Sunday. The right test duration is the maximum of: time to reach sample size, two complete business cycles, and any full marketing cycle (a sale, a launch, a holiday) that could distort behavior.

Probably not on conversion rate. With that traffic, only enormous effects (say, a doubling) are detectable in a reasonable time, and most real product changes deliver 5 to 30 percent lifts at best. Instead of formal A/B testing, run sequential experiments with clear before/after windows, use qualitative interviews to validate changes, and instrument event funnels to catch obvious regressions. Once you cross 5,000 to 10,000 weekly conversions on the page you want to test, statistical testing starts to earn its keep.