Split test calculator

Phrasee Infographic

If you’ve made it this far, you probably know that “statistical significance” is a REALLY important thing to achieve with multivariate testing.

This simple calculator will apply actual statistics to determine how many splits you should use in your tests, and how big each sample size should be.

You’ll also receive an email with more information about subject line testing to help you stay ahead of the marketing curve.

There’s more information about the science behind multivariate testing further down the page but if you wanna get your numbers – go for it. Awesome!



The science behind subject line multivariate testing

OK, here’s a bit of information on proper experimental design…

Email responses follow what’s called a “binomial distribution”. That’s because each result is binary – it’s either opened or not opened, clicked or not clicked, purchased or not purchased.

In the long run, a binomial distribution converges upon a “Gaussian distribution” (a.k.a. “normal” or “bell curve”). This is due to the central tenet of frequentist statistics – the central limit theorem – which means that independent variables (i.e. person 1, person 2, … person 1000 on your list) will, with enough data, converge upon a mean.

From a multivariate testing standpoint, this is important because you need to be able to estimate the minimum amount of data you need to get to mean convergence as reliably as possible.

You also need to consider statistical power – how likely it is that you’re seeing a random run of false positives – and your estimated effect size.

To make matters difficult, not all ESPs make multivariate testing very easy (here’s a guide to what ESPs can test, btw).

In a perfect, clinically controlled world, you’d be able to run infinite tests. But that’s not feasible.

Lastly, there’s the problem of sampling bias. You can’t be 100% certain that your ESP will always randomly select your sample groups. And even if they do, there’s a high probability that there will be inherent skews in your data, thus reducing the reliability of your results.


More opens, more clicks, and more money for you

That’s why we’ve created this split test calculator tool. It’s designed to help marketers in setting up experiments and ensuring robust results.

We’ve helped our customers run literally THOUSANDS of tests over the years and we’ve designed countless scientific experiments. Our Chief Scientist Dr Neil Yager literally wrote the book on data mining. We know what works in email marketing better than anyone else.

And now you can benefit from part of our knowledge.

Phrasee’s end-to-end AI platform generates and optimizes marketing language to increase engagement, conversions, and revenue. Just ask our customers.


Do you want to take your marketing results to the next level?