A/B Testing Sample Size: How to Calculate It

Here's a quick guide to calculating A/B test sample size:

Find your current conversion rate
Choose the smallest change you want to detect
Pick your confidence level (usually 95%)
Decide on test power (typically 80%)
Use a calculator or formula to determine sample size

Key factors affecting sample size:

Current conversion rate
Minimum detectable effect
Confidence level
Test power

Quick sample size guide:

Test Type	Recommended Sample Size
Bayesian	500 per variation
Sequential	1000-1500 per variation
Multi-armed bandit	250 for least performing variation

Common mistakes:

Using too small a sample
Ignoring test power
Forgetting business impact

Remember: Bigger sample sizes can detect smaller differences, but take more time and resources. Balance statistical needs with practical constraints.

Use online calculators or built-in tools in A/B testing platforms for easy sample size calculation.

Basics of Sample Size in A/B Testing

What is Sample Size?

Sample size in A/B testing is the number of users you show each version to. It's crucial for reliable results.

Think of it like a taste test. If only 5 people try your new soda, can you trust their opinion? But if 5,000 people try it, you're onto something.

What Affects Sample Size?

Four main factors impact your sample size:

1. Current conversion rate

Low or high rates? You'll need more people.

2. Smallest change you want to detect

Looking for tiny differences? More people needed. Big changes? Fewer will do.

3. Confidence level

How sure do you want to be? Higher confidence = more people.

4. Test power

This is about catching real differences. More power = more people.

Here's a real-world example:

"If you have a 20% baseline conversion rate and want to detect a 2% increase, the minimum detectable effect (MDE) would be 10%."

Translation: You need enough people to spot a jump from 20% to 22% conversion.

Quick sample size guide:

Test Type	Recommended Sample Size
Bayesian	500 per variation
Sequential	1000-1500 per variation
Multi-armed bandit	250 for least performing variation

But remember, these are just starting points.

A common mistake? Using too few people. Aim for at least 100 conversions per variation.

Grant Tilus, Sr. Growth Product Manager at Cro Metrics, says: "Conversion rate optimization must be a blend of science, math and creativity. Some people are so incredibly math-driven that they tie their own hands."

In other words, numbers matter, but don't forget the human touch. Use tools like AB Tasty, Optimizely, or VWO to calculate your sample size. They'll save you time and help avoid math headaches.

Key Parts of Sample Size Calculation

To get solid A/B test results, you need to nail down four key factors:

Current Conversion Rate

This is your starting point. It's how many visitors take your desired action right now, as a percentage.

Example: If 100 out of 1000 visitors buy your product, your current conversion rate is 10%.

Smallest Change to Detect

How small of a difference do you want to spot? This is your Minimum Detectable Effect (MDE).

Say you want to detect a 2% increase. If your current rate is 10%, you're looking to spot a jump to 12%.

Confidence Level

How sure do you want to be about your results? Most A/B tests use 95% confidence.

This means there's only a 5% chance your results are random.

Test Power

This is about catching real differences when they exist. 80% power is typical.

With 80% power, you'll spot 8 out of 10 real differences between your test versions.

Here's a quick comparison:

Factor	Meaning	Typical Value
Current Conversion Rate	Your starting point	Varies (e.g., 10%)
Smallest Change to Detect	Minimum improvement you care about	2-5% relative change
Confidence Level	How sure you want to be	95%
Test Power	Chance of spotting real differences	80%

These factors work together. Want to detect smaller changes or be more confident? You'll need more test visitors.

"If the conversion rate of variation A is 20% and variation B is 26%, to check statistical significance at a 5% significance level and 80% statistical power, each variation would need 608 visitors, totaling 1216 visitors for the experiment." - MSQRD app case study

How to Calculate Sample Size: Step-by-Step

Let's break down the A/B test sample size calculation process:

1. Find Your Current Conversion Rate

Start with your baseline performance.

Example: 500 buyers out of 10,000 visitors = 5% conversion rate.

2. Choose the Smallest Change to Detect

Decide on your Minimum Detectable Effect (MDE).

Example: Aiming for a 20% relative increase? That's a jump from 5% to 6%.

3. Pick Your Confidence Level

Most A/B tests use 95% confidence. This means a 5% chance of random results.

4. Decide on Test Power

Typical power is 80%. This catches 8 out of 10 real differences between versions.

5. Use a Calculator or Formula

Plug your numbers into a tool. Here's an example using Optimizely's calculator:

Input	Value
Baseline conversion rate	5%
Minimum detectable effect	20% (relative)
Statistical significance	95%
Statistical power	80%

Result: You need 7,990 visitors per variation (15,980 total).

"For a 5% significance level and 80% statistical power, with variation A at 20% and B at 26%, each variation needs 608 visitors. That's 1216 total for the experiment." - MSQRD app case study

Bigger changes need fewer visitors. A 50% boost instead of 20%? You'd only need 1,340 per variation.

Calculating sample size upfront prevents weak tests and time waste. It's your map to solid A/B test results.

Ways to Calculate Sample Size

There are three main ways to figure out sample size for A/B tests:

Using Statistical Formulas

If you're a stats whiz, you can crunch the numbers yourself. But watch out - it's tricky and easy to mess up.

Here's the basic formula for sample size (n) per variation:

n = 2 * (Zα/2 + Zβ)^2 * p(1-p) / d^2

Where:

Zα/2: critical value for significance level
Zβ: critical value for power
p: baseline conversion rate
d: minimum detectable effect

Online Calculators

Want it easy? Use an online calculator. They're quick and simple.

Some popular ones:

Optimizely's Sample Size Calculator
VWO's A/B Test Sample Size Calculator
Evan Miller's Sample Size Calculator

Just plug in:

Your current conversion rate
How big a change you want to detect
How sure you want to be
How much power you need

Let's say you use Optimizely's calculator. You've got a 5% conversion rate now, want to spot a 20% change, with 95% confidence and 80% power. You'd need 7,990 visitors for each version you're testing.

A/B Testing Tools with Calculators

Many A/B testing platforms have calculators built right in. They often give you extra insights, too.

Some examples:

These tools can help you figure out not just how many people you need, but how long your test should run based on your traffic.

"For a 5% significance level and 80% statistical power, with variation A at 20% and B at 26%, each variation needs 608 visitors. That's 1216 total for the experiment." - MSQRD app case study

This shows how your specific test can change the number of people you need.

Here's the deal: Bigger sample sizes can spot smaller differences, but they take more time and resources. You've got to balance what the stats say you need with what you can actually do.

Common Mistakes in Sample Size Calculation

A/B testing success hinges on getting your sample size right. But many testers mess this up. Here are three big no-nos:

Too Small a Sample

This is a rookie error. Small samples can miss real changes. You end up with useless results and wasted time.

"Not enough users in your A/B test? Expect inconclusive results and time down the drain."

Fix: Use the calculators we mentioned earlier. They'll help you nail the right sample size to catch meaningful changes.

Ignoring Test Power

Test power is a big deal, but often forgotten. It's your chance of spotting an effect when there is one. Ignore it at your peril.

Here's what test power means for your sample size:

Test Power	Meaning	Sample Size
80%	Standard	Baseline
90%	Better at catching effects	Bigger
95%	Even more certain	Much bigger

Don't skip test power in your A/B test plans. It's your shield against those annoying "we're not sure" results.

Forgetting Business Impact

Some testers get lost in the numbers and forget why they're testing. A statistically perfect test is useless if it doesn't help your business.

Test changes that matter. Ask:

Will it boost conversions?
Can it increase average order value?
Might it keep customers coming back?

If it's "no" across the board, rethink your test.

Remember: A/B testing is about improving your business, not just crunching numbers. Keep your focus on changes that can actually make a difference.

Changing Sample Size for Different Tests

Your A/B test sample size isn't set in stone. It changes based on what you're testing. Here's how to adjust:

One-sided vs. Two-sided Tests

One-sided tests need fewer people than two-sided tests. Why?

One-sided: Looks for changes in one direction
Two-sided: Checks for changes in both directions

This affects your sample size:

Test Type	Sample Size	Use Case
One-sided	Smaller	When you expect improvement
Two-sided	Larger	When you're unsure

Testing a new button color? If you're sure it'll boost conversions, go one-sided. You'll need fewer people and get results faster.

Testing Multiple Options

Testing more than two options? Bump up your sample size:

Total sample size = Sample size for one option × Number of options

Testing four landing pages instead of two? Double your sample size to keep your confidence level steady.

Seasonal Changes

Seasons can mess with your A/B tests. To fix this:

Run your test longer: Cover a full seasonal cycle
Use a bigger sample: It helps smooth out seasonal ups and downs

An online store might need to test for weeks to account for payday effects on buying.

Don't ignore seasons. They can skew your results. Always think about outside factors that could impact your test.

Using Sample Size Results

Your A/B test sample size is set. Now what? Let's break it down:

What Your Sample Size Means

Think of your sample size as a trust meter for your test results. It's the number of people you need to be confident in your findings.

Let's say you need 2,700 contacts per variation:

You're looking for a 5% shift in your 60% conversion rate
You'll test 27% of your audience
You've got a 95% shot at spotting real changes, not flukes

Here's the deal: More people = More reliable results. But there's a twist.

Practical vs. Statistical Needs

In theory, we'd always use the perfect sample size. In practice? Not so much:

Problem	Solution
Low traffic	Drop confidence level (95% to 90%)
Time crunch	Accept larger minimum effect
Limited resources	Run test longer

Grant Tilus from Cro Metrics nails it:

"Conversion rate optimization is a mix of science, math and creativity. Some folks are so math-focused they limit themselves. They might say, 'Without 99% confidence, we can't act.' But in reality, optimization choices aren't always crystal clear."

The trick? Find the sweet spot between stats and business needs.

Tweaking Sample Size Mid-Test

Sometimes, you've got to pivot. Here's when and how:

1. Seasonal shifts: Testing through holidays? You might need to extend to cover a full cycle.

2. Surprise results: Seeing bigger (or smaller) changes than expected? Adjust your sample size to match.

3. New insights: Fresh info about your audience or product? Time to recalculate.

Just remember: Changing sample size can mess with your results' reliability. Always note why you changed and how it might skew your conclusions.

Conclusion

A/B testing without the right sample size? It's like shooting in the dark. Here's why nailing your sample size matters:

Key Takeaways

1. Sample size = trust

Your sample size directly impacts how much you can trust your test results. Too small? Your data might be off. Too big? You're wasting resources.

2. Find the balance

You need to balance statistical needs with business realities. Sometimes, you'll need to adjust based on traffic, time, or budget.

3. Always calculate

Calculate your sample size before running a test. It helps you:

Avoid weak tests
Spot real differences
Make decisions based on solid data

Next Steps

Ready to put this into action? Here's what to do:

1. Use a calculator

Start with an A/B test sample size calculator. It'll save you time.

2. Set clear goals

Define your minimum detectable effect (MDE) based on your risks and budget.

3. Be patient

Don't cut your test short. Wait for that 95% confidence level.

4. Stay flexible

Be ready to adjust your sample size if needed, but always note why you're making changes.

FAQs

What is a good sample size for AB testing?

For A/B testing, you need at least 1,000 audience members. But that's just the bare minimum.

Want more reliable results? Aim for:

30,000 visitors
3,000 conversions per variant

These numbers help you get results you can trust.

How many participants for AB testing?

It depends on what you're after. Here's a quick breakdown:

Reliability	Minimum Visitors	Minimum Conversions per Variant
Basic	1,000	100
High	30,000	3,000

Keep in mind: These are just guidelines. Your specific needs might be different based on your conversion rates and what you're testing.

How to calculate required sample size for AB test?

Here's how to figure out your A/B test sample size:

Check if you have enough contacts
Use an A/B test calculator
Input your current conversion rate
Enter your minimum detectable effect
Look at the results
Figure out what percentage of your audience this sample size represents

Don't want to do the math? No problem. There are plenty of online tools that'll do it for you.

How to calculate AB testing sample size?

Want to get your hands dirty? Here's the basic formula:

n = sample size
p1 = current conversion rate
p2 = expected new conversion rate
Zα/2 = Z-score for your confidence level

But let's be real: The actual formula is pretty complex. That's why most folks use calculators or built-in tools in A/B testing platforms.