04 Jun 2015
Can you predict which email subject line won this split test?
Phrasee uses artificial intelligence to put science behind subject line language. The technology we use – Natural Language Generation – is nascent… and lots of people don’t believe it can work. Seriously! We guess people have a hard time believing that computers can generate better marketing language than humans.
It’s hard to predict which email subject line won a split test. Trust us.
So, we thought we’d put ourselves to the test. One of our clients – a major UK airline – at first didn’t believe that Phrasee’s robot overlord could generate better email subject lines than humans could.
They sure believe we can now. And here’s why.
Can YOU predict which email subject line won?
The best way to test things out was… well, to run a test! We used one of their subject lines and four of Phrasee’s generated subject lines.
Here are the five subject lines we tested. We’ve redacted the client name at their request. They don’t want everyone knowing their secret to subject line success!
- Wow. Just wow. Get Caribbean holidays starting at just £599…
- An elite offer for exclusive Caribbean holidays.
- ➢ Discover why BRAND NAME is where to buy Caribbean holidays!
- ➾ Hope you’re sitting down.
- Bargain alert! BRAND NAME’s sale just got bigger.
To add some context: the goal of the campaign was to convert subscribers who hadn’t purchased anything before. Each subject line was sent to a sample group of 4238 people, as calculated by Phrasee’s in-app sample size calculator. Then, the ‘winner’ was sent out to everyone else 24 hours later. The winner was determined using a weighted combination of opens, clicks and revenue generated.[poll id=”2″][poll id=”3″]
Let’s take a look at the composite parts of the subject lines.
The subject lines were 61, 49, 62, 28 and 50 characters long, respectively. But this means nothing, because subject line length is not an indicator of success. Trust us. So forget that.
All of the subject lines use plausible English and “sound” human.
Here’s some more differences:
- Two of them use symbols
- Two of them use exclamation marks
- One uses a descriptive adjective
- One mentions price
- Two mention a sale or offer is on
- Two use a second-person imperative verb conjugation
- Two are broken up into multiple sentence fragments
- One ends in ‘dot dot dot’
- And so on…
Predicting results based upon disparate variables as above is impossible. None of the individual elements operate in isolation. Each of the words, grammatical structures and symbols interact with each other. This high-order interaction makes determining causal variables beyond the reach of human brainpower.
Humans interpret language holistically. That means: you don’t read one word, then the next, and then the next. You read the entire chunk of text as a whole and react to it emotionally. (It’s science.)
To quantitatively measure the emotional elements of the subject lines, we ran the subject lines through Phrasee Pheelings™. It’s our proprietary marketing emotion analysis tool. Here’s what its output was for these five lines:
Phrasee Pheelings™ analyses hundreds of emotional sentiments, and then scores the subject lines on each of them.
So now the question to ask is no longer “Which words drive the best response?”
The question to ask is: “Which combination of words and emotions drive the best response?”
Here’s which subject line was written by a human
Subject line #2 was the only one written by a human in our client’s email team (“An elite offer for exclusive Caribbean holidays.”)
Bet the results surprised you! Artificial Intelligence can generate human-sounding language. It’s a huge trend in marketing right now.
But, how well does machine-generated language work? Let’s find out…
Here’s which subject line won the split test
Any of the five subject lines are plausible winners. They could all win. But only one did.
And the winner is…
Here’s the actual open rate, click through rate, and click-to-open-ratio per subject line:
|Wow. Just wow. Get Caribbean holidays starting at just £599…||19.6%||2.9%||14.8%|
|An elite offer for exclusive Caribbean holidays.||13.4%||1.8%||13.4%|
|➢ Discover why BRAND NAME is where to buy Caribbean holidays!||15.8%||2.3%||14.6%|
|➾ Hope you’re sitting down.||18.2%||2.1%||11.5%|
|Bargain alert! BRAND NAME’s sale just got bigger.||12.6%||1.4%||11.1%|
The first one in the list was the best subject line in terms of both opens and clicks (and conversions too – but our client was loathe to share their exact commercials in public.)
Using the winning subject line to send to everyone else on their list resulted in a 46% lift in opens, and a 61% lift in clicks.
Not too bad for a computer, hey?
Why did that subject line win?
The first subject line was the clear winner. Why? You could list off hundreds of reasons, including:
- Using sentence fragments is good
- The word ‘wow’ is good
- Mentioning the price at the end is good
- Using a direct verb ‘get’ is good
- Using symbols is bad
- Mentioning the starting price is good
- Or anything else…
And you could argue compellingly for any of the above points. But – and this is an important point – it wasn’t just one of the factors. It was the higher-level interaction of all of the language elements that actually drove response.
Using Phrasee Pheelings™ we created emotional clusters to quantitatively measure the overall emotional effect of subject lines.
The simple fact is this: marketing language works when it hits the right emotional triggers for your audience. You can only get so far with ‘gut instinct.’
That’s why we built Phrasee Pheelings™.
Quantitatively measuring emotional response to language – and learning from it to power future campaigns – is a near-impossible task for humans.
But using artificial intelligence it is a solvable problem.
To find out more about how Phrasee Pheelings™ can drive engagement for your brand, book a demo with one of our subject line scientists. You’ll be surprised what you learn.