Email subject line length doesn’t affect open rates – but most people who say it use bad statistics
Here at Phrasee we love proving or disproving hypotheses using statistics. It’s fun. In a very nerdy sort of way.
There have been a few blog posts lately about email subject line length.
This particular post’s thesis was correct! Subject line length has a very tiny (if any) effect on response rates.
But their methodology was flawed. They guessed correctly and got lucky… and risked misleading a lot of people with dodgy statistics.
There’s been a bunch of other blogs about this topic over the years with even more flawed methodology. We were going to link to them but it made us feel dirty. No backlinks for bad statistics.
Not to fear! Phrasee is here to correct the statistical shortcomings of those around us.
How was their methodology faulty?
Their research didn’t normalise their statistics beforehand.
What’s that mean? Let us explain.
Let’s say we have two people – let’s call them Phreddy and Phrancesca.
Phreddy always sends out emails with short subject lines (e.g. 20 characters) and averages a 25% open rate.
Phrancesca always sends out emails with long subject lines (e.g. 80 characters) and averages a 5% open rate.
Based on this, a naïve interpretation is that short subject lines are better than long ones. However, what if I tell you a little secret: Phrancesa sells frankfurters, and Phreddy sells flowers. That changes things, doesn’t it! Subject line length is just one of many different factors that impact open rate, so it’s unfair to directly compare Phreddy’s subject lines with Phrancesca’s.
The idea behind “normalisation” is to transform the open rates so that they can be compared with each other. What you should do is compare Phreddy’s results with Phreddy’s, and Phrancesca’s with Phrancesca’s. Then, look at the relative lift that short or long subject lines have within their own data sets.
Think about it like this: you shouldn’t compare frankfurters and flowers because you aren’t going to learn anything.
And another thing! (Get these damn kids off my lawn…)
Another area where people go wrong has to do with sample size, variability and over-fitting. Let me explain.
I’ve actually seen people, in all honesty, show me graphs like this:
And say “Wow! Subject lines of length 40 and 220 are great, but 70 and 250 characters are terrible! Who would have guessed! Oh well, numbers don’t lie.”
Of course that’s not what’s going on. That’s ridiculous.
What is going on? It comes down to sample size. Most subject lines have a medium length, and really short and really long subject lines are rare. When you don’t have many examples, the average rate is very sensitive to outliers and you don’t get a reliable estimate. Therefore, you can’t just rely on averages. You also need to consider how much confidence you have in an estimate.
No, I will not calm down! (Why are these damn kids still on my lawn?)
We’ve got some charts to help illustrate things.
This is what bad statistics will tell you about email subject line length
We took a bunch of anonymised Phrasee user data and graphed the effect that subject line length has on non-normalised open rates. We used Python’s native output, so that’s why the graphs look pretty scientific. It looks like this:
What most “analysis” of email subject line length does is they only look at the blue line, for non-normalised data. And this leads you up the garden path.
What it would tell you is that short subject lines are good, middle-length ones are bad, and long ones are good.
And this simply isn’t the case.
Note two things: this data isn’t normalized, and you can see that we have less confidence for really short and long subject lines.
We ran the same statistics as in the blog post we mentioned at the start of this article, and got the following chart:
(Correlation coefficient of -17.66%… indicating a strong inverse relationship…)
Only use short subject lines! But that would be a bad idea. Trust us. We know a thing or two about email subject lines.
Here’s bad statistics for the number of words in a subject line
(Correlation coefficient of -19.05%)
Here’s bad statistics for the average word length in a subject line
(Correlation coefficient of +5.19%)
These charts collectively tell you nothing about email subject line length. What they do tell you is that using flawed methodology can prove anything incorrectly.
Here’s good statistics for email subject line length
We took the same anonymised Phrasee user data as before and graphed the effect subject line length has on normalised open rates. It looks like this:
There is a *tiny* effect. But the light-blue confidence intervals indicate this is just due to random variance in the sample set.
This proves that for any given subject line length, it is possible to have high open rates or low open rates. The impact of subject line length has a very weak, if any, impact on open rate.
Now it’s time to get gangsta… (Those damn kids on my lawn better watch out…)
Just to prove the point again… we used another methodology to make sure we were right (spoiler alert: we were.)
We ran a regression analysis to determine the relationship between open rates and subject line length
Warning: this section contains LOTS of detail. LOTS. And lots of advanced statistics. Don’t worry if you don’t understand it. We’re just ranting at this stage because we hate flawed statistical methodology so much.
Regression is a statistical method to determine how variables affect a result. In this case, we were looking to determine what effect 1) subject line length; 2) the number of words in a subject line; and 3) the average word length have on open rates.
EDIT: It’s been pointed out to us that this is a massive simplification of predictive analytics. You’re right! Our point is to simply show that *any* methodology using normalised statistics gives a better result. There are other methodologies that are perhaps more appropriate (for example General Linear Regression, as one particular troll very rudely pointed out) and if you want to have a crack, we’ll buy you dinner (KFC or BK, your choice) if the results are (robustly) different than ours.
The point remains that using normalised statistics gives you a much better indication of what works and what doesn’t. The take from this blog post is that any statistics that don’t use normalised means are methodologically dodgy and shouldn’t be trusted.
Short answer: barely any effect.
Next, we ran the regression using this formula:
Y = µ+ β1*X1 + β2* X2 + β3*X3
- Y = the calculated open rate
- µ = your average open rate
- X1,2,3 = subject line length, # of words, and average word size, respectively
- β1,2,3 = the calculated coefficient from our regression model
Here’s what we discovered:
Subject line length accounts for just 0.1% of email open rate variance
Here’s the full regression output:
The “Adjusted R Square,” aka the Coefficient of Determination, indicates that this is not a good predictive model for email subject line effectiveness. 0.001393685 is low. Very low.
It can be interpreted as: “These variables don’t tell you anything about anything.”
Our not-so predictive-but-statistically-robust formula is:
Open rate = (Average open rate) + 0.018*Length + -0.139*#words + -.067*AvgWordLength
Try and plug in your own numbers. It’s an effect so small Sherlock Holmes wouldn’t be able to find it. Other factors affect open rates much more (find out more about what optimises email subject lines here.)
The only other research we could find that used good methodology, from way back in 2012, backs up our results.
So there you go.
TL;DR – email subject line length has no effect on open rates
The thesis in the original blog post that spurred our research was correct. Subject line length has pretty much no effect on open rates. But their statistical methodology was flawed. They got lucky this time.
So here’s some free advice – when you see interesting statistics about email subject lines, make sure they are statistically significant…
Because bad statistics cause bad decisions.
It’s like a jungle out there – so be careful!
PS – we use statistics like the preceding analysis to optimise email subject lines. Next step? Book a demo obviously!