A/B testing for subscription stores, without lying to yourself

What's worth testing — and what isn't

The single biggest mistake is testing things that don't matter. Button colors on the subscribe widget have been tested to death by twenty years of e-commerce research and the effect size is in the low single digits — too small to detect at subscription-store sample sizes, and too small to matter even if detected. Save your testing budget for changes large enough to move a metric you can measure in a reasonable time.

The high-leverage experiments cluster in five areas: subscription discount level, widget copy and framing, cancel-flow save offer, dunning email content, and pricing-page layout for subscription tiers. Each of these can produce 10-30% relative shifts in the relevant metric, which is large enough to detect with realistic subscription sample sizes and large enough to matter to revenue.

• Subscription discount level (5% vs 10% vs 15%) — moves subscribe-rate AND lifetime margin, often in opposite directions. The most consequential test you can run.
• Widget copy and framing ('Subscribe & Save 10%' vs 'Auto-deliver every month' vs 'Never run out') — moves opt-in rate measurably.
• Cancel-flow save offer (pause vs discount vs skip-next) — directly moves churn; results visible within weeks.
• Dunning email cadence and content — moves involuntary-churn recovery; measurable within 30 days.
• Cadence default (monthly default vs every-other-week default) — affects sign-up rate and AOV simultaneously.
• Pricing-page tier layout — anchoring, tier order, recommended badge placement all measurably affect mix.

Sizing the test: how many subscribers do you actually need

Sample-size math is the part most merchants skip and then regret. The honest answer depends on three numbers: your baseline conversion rate (or whatever metric you're measuring), the minimum effect size you want to detect, and the variance of the outcome. A 5% relative lift on a 3% subscribe-rate baseline takes a few thousand visitors per variant. A 5% relative lift on a 0.5% baseline takes tens of thousands. Most subscription stores don't have that kind of traffic on a single PDP.

There's a simple rule of thumb that gets you close. For a binary outcome (subscribed yes/no, cancelled yes/no) at typical subscription rates, you need roughly 16 × (1/p) × (1/MDE²) visitors per variant, where p is your baseline rate and MDE is the minimum detectable effect as a decimal. For a 3% baseline and a 10% relative lift (MDE = 0.003), that's 16 × 33 × 111 ≈ 58,000 visitors per variant. For a 6% baseline and the same lift, it's 16 × 17 × 111 ≈ 30,000. The formula is approximate but it'll tell you whether your idea is feasible before you start.

If the answer is 'we don't have that traffic', you have three options: run the test longer (months not weeks), pick a coarser MDE (look for 25% effects, not 10%), or pick a higher-traffic surface (the home page, not a single PDP). Don't run an underpowered test and read the result — you'll get a coin flip and call it a win.

Statistical significance for non-statisticians

Significance testing answers one question: how likely is it that the difference between variants happened by chance rather than because the change actually worked. The conventional threshold is p < 0.05, meaning there's less than a 5% chance of seeing this gap if the variants were truly identical. That's an arbitrary cutoff but it's the industry norm and it's a reasonable starting point.

Three caveats that subscription merchants get wrong constantly. First, p < 0.05 doesn't mean 'this works' — it means 'this difference is unlikely to be random.' The effect could still be small or the test could be biased by a confounder. Second, peeking at results daily and stopping when significance is reached inflates your false-positive rate dramatically — you can hit p < 0.05 by chance just by checking often enough. Decide your sample size up front and only read the result at the planned endpoint. Third, conversion lift doesn't equal revenue lift. A variant that lifts subscribe rate by 20% but increases churn by 30% loses money — but that won't show up in a 14-day conversion test.

• Pre-register the test — write down sample size, duration, primary metric, and stopping rule before launch
• Don't peek — checking results daily and stopping when significance is reached inflates false-positive rate from 5% to 25%+
• One primary metric — secondary metrics are interesting but the test is decided on the primary one alone
• Bayesian or frequentist, pick one — both work, mixing them mid-test does not
• Sanity check — if the lift looks too good (50%+ on a content tweak), it's probably a tracking bug

The proxy-metric trap: conversion vs lifetime value

The metric that pays your bills is lifetime value — total margin per subscriber across the whole subscription life. The metric you can measure quickly is conversion (visitors who subscribe) or short-run retention (subscribers who survive 30 days). These often correlate, but not always, and the cases where they diverge are exactly the cases where A/B testing leads you astray.

Classic example: a deeper subscription discount lifts conversion (more visitors subscribe because the offer is sweeter) but pulls in price-sensitive subscribers who churn faster. The conversion test says 'ship it.' The cohort retention curve 6 months later says 'we just gave away 5 points of margin for nothing.' Always pair a conversion test with a retention readout, even if the retention data takes longer to arrive.

The practical workaround: run the test, ship the winner if conversion clearly wins, but tag the cohort that experienced each variant and check retention at month 3 and month 6. If the conversion-winning variant retains worse, roll it back. This is messier than a clean A/B test but it's the only honest way to read subscription experiments when the true metric is months away. Our analytics dashboard guide walks through cohort retention reading in detail.

Pitfalls specific to subscription A/B testing

Standard A/B testing wisdom assumes independent samples, stable populations, and outcomes that materialize quickly. Subscription contexts violate all three in subtle ways that produce false wins.

• Returning subscribers contaminate the test — a logged-in subscriber who hits the PDP isn't a fresh sample; they already subscribed. Exclude active subscribers from PDP conversion tests.
• Renewal-day spikes skew weekly numbers — if your renewal day is the 1st and your test starts on the 28th, the first 3 days are unrepresentative. Run tests across at least 2 full renewal cycles.
• Seasonality bites harder — gift subscriptions, holiday gift boxes, January resolutions all distort subscriber behavior. A test that runs October-November is measuring something different than the same test in February.
• Save-flow tests confound with reason — a save offer that wins on 'too expensive' cancellers might lose on 'got too much' cancellers. Segment by cancel reason when reading save-flow tests.
• Mobile vs desktop divergence — many widget changes win on desktop and lose on mobile (or vice versa). Segment by device or your overall result is an average that doesn't apply anywhere.
• The novelty effect — any new save flow looks better at first because subscribers haven't learned the patterns yet. Wait 2-3 cycles before reading.

Designing the experiment: a worked example

Walk through a realistic example. You suspect changing the widget headline from 'Subscribe & Save 10%' to 'Never run out — auto-deliver every month' will lift the subscribe rate. Baseline subscribe rate from PDP visitors is 4%. You'd consider a 15% relative lift worth shipping (subscribe rate goes to 4.6%). PDP traffic is ~2,000 visitors per week.

1. Compute required sample — 16 × (1/0.04) × (1/0.006²) ≈ 110,000 visitors per variant. At 1,000 visitors/week/variant that's 110 weeks. Not feasible.
2. Re-scope — raise the MDE to 25% relative lift (effect of 0.01). Sample drops to ~16,000 per variant. At 1,000/week/variant that's 16 weeks. Still long.
3. Pick a higher-traffic surface — run the test across all subscription-eligible PDPs (8,000/week total). That gets you to a 4-week test for a 25% MDE, which is workable.
4. Pre-register — primary metric: subscribe rate. Secondary: AOV, 30-day retention. Stopping rule: 4 weeks regardless of significance.
5. Run the test — 50/50 split, don't peek at significance daily.
6. Read at endpoint — variant subscribe rate 4.7%, control 4.0%, p = 0.04. Significant.
7. Ship, then watch retention — tag the cohort and re-check month-3 retention vs the prior cohort. If retention holds, the win is real.

Notice how much of the work happens before any code change. The 'experiment' is really a decision about what's worth measuring. Once you've made that decision honestly, the test itself is mechanical.

Five experiments worth running first

If you're starting a testing program from scratch, these five tend to produce the largest, fastest, most reliable wins for subscription stores. Run them in roughly this order.

1. Discount level test (5% vs 10% vs 15%) — move both subscribe rate AND retention; the most consequential single test you can run.
2. Widget headline reframe ('Save 10%' vs 'Never run out' vs 'Auto-deliver') — easy to set up, often surprising winners.
3. Cancel-flow save offer (pause vs 1-month discount vs 'skip next delivery') — directly moves churn; readout within weeks.
4. Dunning email subject line ('Update your payment method' vs 'Your next delivery is at risk') — easy win on involuntary churn recovery.
5. Cadence default on widget (monthly preselected vs every-other-week preselected) — moves AOV and retention simultaneously; often surprises merchants.

When to stop testing and just ship

Testing has overhead. Engineering time, analyst time, the opportunity cost of running a 4-week test instead of just shipping the better-looking variant. There are times when you should skip the test entirely and ship the change.

• The current state is clearly broken — fix it, don't test it. Don't A/B test a bug.
• The change has external evidence — if 5 similar stores have tested this and the result is consistent, you're not learning anything by re-testing.
• Sample size is impossible — if your traffic can't produce a meaningful result in 12 weeks, the test will tell you nothing. Pick a heuristic-driven choice and move on.
• The change is reversible and small — try it for 2 weeks, watch the dashboard, roll back if it gets worse.
• You already know the answer — if you're testing to confirm a hypothesis you'd ship anyway, you're wasting the time. Just ship.

The corollary: when the change is consequential, hard to reverse, and you genuinely don't know which way it'll go — that's when an A/B test pays for itself. Discount-level changes, save-flow rework, pricing-page restructure. The rest of the time, ship the change with good monitoring and roll back if the dashboard says you got it wrong.