Our Blog

“Knowledge is knowing a tomato is a fruit. Wisdom is not putting it in a fruit salad.”


  • Categories

Search Blog

We ran a subject line split test and learned nothing and everything at the same time

Here at Phrasee, we’re all about testing. This isn’t news. We had a hunch, and ran a subject line split test in our newsletter last week. The results will boggle your mind…

Here’s the results:

Version A Version B
Open % 33.9% 31.2%
Click % 6.2% 6.3%
CTOR % 18.3% 20.3%


Overall, A delivered a 9% lift on open rates. Pretty good result!

And B drove an 11% lift in CTOR. Amazeballs!

Being the statistics nerds we are, we then checked for confidence, aka statistical significance.

Opens: 90% confidence. 90% is a reasonably high level of confidence.

Clicks: 54% confidence. Negligible result.

CTOR: 85% confidence. That’s fairly confident.

[poll id=”4″]

If you haven’t signed up for Phrasee’s weekly newsletter, do it now. It just may change your life*.
*No lives will be changed.

So what was in our subject line split test?

We tested nothing.

We randomly split our list in half and sent out the exact same email with the exact same subject line to both groups.

Here’s what we learned: we are 100% confident that random variance exists.

Hello, counter-intuition, is it me you’re looking for?

With 90% certainty version A is better than version B. But version A was the exact same as version B.

What’s going on here?

When you look at one split test and make a decision based upon it, even when using statistically robust methods, you could be falling victim to random variance.

Random variance occurs when unknown or uncontrollable variables affect your results. And email subject lines are prone to random variance at a high level. That’s why this split test, while 90% statistically significant, doesn’t tell you a whole lot.

Longitudinal learning overcomes random variance


When you run one split test and assume its result is inherently true, you are risking making a bad decision. Smart marketers look at longitudinal results.

Longitudinal learning occurs when you test multiple things over time and, when analysing it, control for random variance.

This is what Phrasee does. Using machine learning, we take all of your split test results and algorithmically remove random variance from the equation.

Don’t be a victim of random variance.

Book a demo with Phrasee to learn how you can actually learn from your split tests.

Trevor Beers, Senior Language Technician, Phrasee

Phrasee pheatures: Trevor Beers

Our very own Senior Language Technician, Trevor Beers, shares advice for those looking to join the tech industry, and insights into life at Phrasee.

Discover how applying actual AI to language can deliver big results!