29 Aug 2017
The awesome Phrasee split test calculator
Hey! If you’ve made it this far, then you probably know that “statistical significance” isn’t just the strongest Tinder opener ever – it’s also a REALLY important thing to achieve when split testing.
This simple calculator will use actual statistics to determine how many splits you should use, and how big each sample size should be.
You’ll also get triggered an email with a bunch more information about subject line testing. So that’s cool.
There’s more info further down the page if you’re interested. But if you just wanna get your numbers – go for it. Awesome!
Here’s a bunch more info:
OK, here’s a bit of information on proper experimental design. It’s sort of academic because what you ultimately care about is results. But hey, in case you’re interested, read on…
Email responses follow what’s called a “binomial distribution”. That’s because each result is binary – it’s either: opened or not opened, clicked or not clicked, purchased or not purchased.
In the long run, a binomial distribution converges upon a “Gaussian distribution” (aka “normal” aka “bell curve”). This is due to the central tenet of frequentist statistics – the central limit theorem – which means that independent variables (i.e. person 1, person 2, … person 1000 on your list) will, with enough data, converge upon a mean.
From a split testing standpoint, this matters… and here’s why: you need to be able to estimate the minimum amount of data you need to get to mean convergence as reliably as possible.
You also need to consider things like statistical power – how likely it is that you’re seeing a random run of false positives – and your estimated effect size.
To make matters more difficult, not all ESPs make split testing very easy (here’s a guide to what ESPs can split test, btw).
In a perfect, clinically controlled world, you’d be able to run infinite split tests. But that’s not feasible.
Lastly, there’s the problem of sampling bias. You can’t be 100% certain that your ESP will always randomly select your sample groups. And even if they do, there’s a high probability that there will be inherent skews in your data, thus reducing the reliability of your results.
That’s why this email split testing calculator is awesome.
We’ve helped our clients run literally THOUSANDS of split tests over the years. We know what works in email marketing, probably better than anyone else.
We’ve done the stats. We’ve designed countless scientific experiments. And we know what we’re talking about. Oh yeah, our Chief Scientist Dr Neil Yager literally wrote the book on data mining. So, yeah, you can probably believe what the handsome devil says.
And now you can benefit from part of our knowledge.
BTW – setting up your test is only half the battle – you also need to figure out what you want to actually achieve. If you want more opens, more clicks, and more conversions… well, that’s what we do. Just ask our customers.
Get in touch if you want to take your email results to the next level!