How much traffic does a Shopify A/B test need?

Most Shopify A/B tests are decided by noise. A new product page template runs for ten days, the dashboard shows a 12% lift with a green badge, the change ships, and three months later conversion is where it was. Usually nothing was wrong with the testing tool. The test simply never had enough traffic to detect the effect it was looking for. Working out the traffic you need before you start is a five-minute calculation, and it changes which tests are worth running at all.

What is power analysis and why does it matter?

Every A/B test can go wrong in two ways:

• False positive (type I error): declaring a winner when there is no real difference. Its probability is the significance level, α, conventionally 5%.
• False negative (type II error): missing a real improvement. Power is the probability of avoiding it, conventionally 80%.

Power analysis connects four quantities: significance level, power, the baseline metric, and the minimum detectable effect (MDE), the smallest true lift you want the test to catch. Fix three and the fourth follows. In practice you fix α, power and the baseline, choose the MDE, and solve for the sample size.

The MDE is a business decision, not a statistical one. Ask: what is the smallest improvement that would justify shipping and maintaining this change? If a 3% lift would not change anything you do, do not design a test to detect it.

How do you calculate the sample size for a conversion rate test?

For comparing two conversion rates p₁ (control) and p₂ (variant) with equal traffic, a standard approximation for the visitors needed per variant is:

n = (z₁₋α/₂ + z₁₋β)² × [p₁(1 − p₁) + p₂(1 − p₂)] / (p₂ − p₁)²

where z₁₋α/₂ = 1.960 for a 5% two-sided test and z₁₋β = 0.842 for 80% power.

A worked example

A store converts 2.0% of visitors and wants to detect a 10% relative lift, from 2.0% to 2.2%.

1. (z₁₋α/₂ + z₁₋β)² = (1.960 + 0.842)² = 2.802² ≈ 7.849
2. Variance term: 0.02 × 0.98 + 0.022 × 0.978 = 0.01960 + 0.02152 = 0.04112
3. Squared difference: (0.022 − 0.020)² = 0.002² = 0.000004
4. n = 7.849 × 0.04112 / 0.000004 ≈ 80,700 visitors per variant, about 161,400 in total.

You can check this with statsmodels, which uses Cohen’s h (an arcsine transformation of the two proportions) as the effect size:

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

h = proportion_effectsize(0.022, 0.020)
n = NormalIndPower().solve_power(effect_size=h, alpha=0.05, power=0.8,
                                 ratio=1.0, alternative="two-sided")
print(round(n))  # 80637 per variant

The two methods agree to within a fraction of a percent. Evan Miller’s free online sample size calculator takes the same inputs (baseline conversion rate, minimum detectable effect, power and significance level) if you prefer not to write code.

How the answer moves with the MDE and baseline

Visitors needed per variant, α = 5% two-sided, power = 80%:

Baseline conversion	Relative lift to detect	Visitors per variant
2%	5% (2.0% → 2.1%)	~315,200
2%	10% (2.0% → 2.2%)	~80,700
2%	20% (2.0% → 2.4%)	~21,100
2%	30% (2.0% → 2.6%)	~9,800
3%	10% (3.0% → 3.3%)	~53,200

Two patterns stand out. Halving the MDE roughly quadruples the traffic, because the difference enters the formula squared. And a higher baseline conversion rate needs less traffic for the same relative lift.

To turn that into a duration, divide the total by weekly visitors to the tested page, not to the whole store, and round up to whole weeks. A hypothetical store with 25,000 weekly visitors to product pages would need about 161,400 ÷ 25,000 ≈ 6.5, so seven weeks, to detect a 10% lift from a 2% baseline. Running whole weeks keeps weekday and weekend shoppers balanced.

Read the table the other way when traffic is fixed: with 10,000 visitors per variant at a 2% baseline, the smallest lift you can reliably detect is about 30%. Few page changes move conversion by that much.

Why is peeking at results a problem?

The 5% false-positive rate assumes you look at the result once, at the planned sample size. Most testing dashboards update continuously, and the temptation is to stop as soon as the variant shows as significant.

Evan Miller’s “How Not To Run an A/B Test” shows why that breaks the math: random fluctuations cross the significance threshold at some point far more often than at one pre-chosen point. In his example, continuous monitoring with stopping at the first significant result pushes the real false-positive rate from a nominal 5% to 26.1%.

The fixes:

• Decide the sample size in advance and do not act on results until you reach it.
• If you need to look early, use a sequential testing method designed for interim analyses, which adjusts significance thresholds for each look. Check that your testing tool actually implements one before relying on its early-stopping advice.

Stopping early for a large negative effect, such as a broken checkout, is a different matter and is fine; that is a quality decision, not an inference.

How do novelty effects distort Shopify tests?

Returning customers notice change. A redesigned navigation or a new product page layout can attract extra clicks at first simply because it is new, or depress behavior briefly while people relearn where things are. Either way, the first week’s effect is not the long-run effect.

Practical checks:

• Plot the treatment effect by week. A lift that shrinks steadily is a warning sign.
• Split results by new vs returning visitors. New visitors have no prior experience to react to, so their result is closer to the steady state.
• Do not cut a test short just because the early lift looks large; that is exactly when novelty and noise are most likely to be inflating it.

Should you test conversion rate or revenue per visitor?

Conversion rate tells you whether more visitors bought. Revenue per visitor (RPV), revenue divided by visitors including the ones who did not buy, tells you whether the store made more money. A change that nudges conversion up while pushing shoppers toward cheaper products can lose money, and only RPV sees that. For pricing, bundling, free-shipping thresholds and upsells, RPV is the right decision metric.

The cost is noise. Each visitor’s revenue is zero most of the time and a variable order value otherwise. If p is the conversion rate and CV is the coefficient of variation (standard deviation divided by mean) of order value, the squared coefficient of variation of revenue per visitor is:

CV²(RPV) = (1 + CV²(order value)) / p − 1

For conversion rate alone it is (1 − p) / p. Since sample size is proportional to CV², RPV needs roughly (1 + CV²(order value)) times the traffic of a conversion test for the same relative lift. When order values have a standard deviation about equal to their mean (CV = 1), that is about twice the traffic; with more variable baskets it is more.

Two practical points for RPV tests:

• Cap extreme orders (winsorize at a high percentile decided before the test) so one wholesale-sized order does not decide the result.
• Exclude or separately handle B2B and staff orders, which can land in either variant by chance.

What should you do when traffic is too low?

Many Shopify stores cannot reach 80,000 visitors per variant on any single page in a reasonable time. Options, roughly in order of usefulness:

1. Test bigger changes. A new offer, pricing structure or page concept can plausibly move conversion by 20% or more; button colors cannot. Bigger expected effects need far less traffic.
2. Test where the traffic is. Sitewide elements (header, cart, shipping messaging) see every visitor; a single product page does not.
3. Use a metric closer to the change, such as add-to-cart rate for a product page change. It has a higher baseline and responds more directly, so it needs less traffic. Treat a win as directional evidence, not proof of more revenue.
4. Reduce variance with pre-experiment data. CUPED (Deng, Xu, Kohavi and Walker, 2013) adjusts the metric using each user’s behavior before the test. On Bing’s experiments the authors report reducing variance by about 50%, equivalent to the same power with half the users. It helps most when many visitors are returning customers with useful history.
5. Run fewer variants. Every extra variant splits traffic further; A/B beats A/B/C/D on a small store.
6. Decide without a test. For low-risk changes backed by user research, shipping and monitoring a before/after trend can be a reasonable choice, as long as everyone understands it is weaker evidence than a controlled test.

What not to do: lower the bar quietly by stopping early, raising α after the fact, or slicing results into segments until one looks significant. Each of those turns a test into a source of confident wrong answers.

How should a Shopify team plan its next test?

Before launch, write down the primary metric, the MDE, the sample size, the planned duration in whole weeks and the decision you will make at each outcome. If the calculation says the test cannot detect any effect you would care about within a couple of months, change the test, not the statistics.

Asabisoft helps Shopify brands set up this kind of experiment design and analysis when testing decisions start to carry real budget; the founder has run funnel A/B testing for Shopify stores and A/B testing and monetization analysis at N3TWORK Studios. For measuring the long-run value of the customers a winning variant brings in, see our guide to cohort retention analysis.