Email subject line length doesn’t affect open rates – but most people who say it use bad statistics

Here at Phrasee we love proving or disproving hypotheses using statistics. It’s fun. In a very nerdy sort of way.

There have been a few blog posts lately about email subject line length.

This particular post’s thesis was correct! Subject line length has a very tiny (if any) effect on response rates.

But their methodology was flawed. They guessed correctly and got lucky… and risked misleading a lot of people with dodgy statistics.

There’s been a bunch of other blogs about this topic over the years with even more flawed methodology. We were going to link to them but it made us feel dirty. No backlinks for bad statistics.

Not to fear! Phrasee is here to correct the statistical shortcomings of those around us.

How was their methodology faulty?

Their research didn’t normalise their statistics beforehand.

What’s that mean? Let us explain.

Let’s say we have two people – let’s call them Phreddy and Phrancesca.

Phreddy always sends out emails with short subject lines (e.g. 20 characters) and averages a 25% open rate.

Phrancesca always sends out emails with long subject lines (e.g. 80 characters) and averages a 5% open rate.

Based on this, a naïve interpretation is that short subject lines are better than long ones. However, what if I tell you a little secret: Phrancesa sells frankfurters, and Phreddy sells flowers. That changes things, doesn’t it! Subject line length is just one of many different factors that impact open rate, so it’s unfair to directly compare Phreddy’s subject lines with Phrancesca’s.

The idea behind “normalisation” is to transform the open rates so that they can be compared with each other. What you should do is compare Phreddy’s results with Phreddy’s, and Phrancesca’s with Phrancesca’s. Then, look at the relative lift that short or long subject lines have within their own data sets.

Think about it like this: you shouldn’t compare frankfurters and flowers because you aren’t going to learn anything.

Say it with sausage

And another thing! (Get these damn kids off my lawn…)

Another area where people go wrong has to do with sample size, variability and over-fitting. Let me explain.

I’ve actually seen people, in all honesty, show me graphs like this:

bad email subject line length analysis

And say “Wow! Subject lines of length 40 and 220 are great, but 70 and 250 characters are terrible! Who would have guessed! Oh well, numbers don’t lie.”

Of course that’s not what’s going on. That’s ridiculous.

What is going on? It comes down to sample size. Most subject lines have a medium length, and really short and really long subject lines are rare. When you don’t have many examples, the average rate is very sensitive to outliers and you don’t get a reliable estimate. Therefore, you can’t just rely on averages. You also need to consider how much confidence you have in an estimate.

No, I will not calm down! (Why are these damn kids still on my lawn?)

We’ve got some charts to help illustrate things.

This is what bad statistics will tell you about email subject line length

We took a bunch of anonymised Phrasee user data and graphed the effect that subject line length has on non-normalised open rates. We used Python’s native output, so that’s why the graphs look pretty scientific. It looks like this:

subject line length bad statisticsNote that the solid blue line is the actual result, and the light blue area is the “confidence interval” – which means where the results lie when taking into account statistical variance.

What most “analysis” of email subject line length does is they only look at the blue line, for non-normalised data. And this leads you up the garden path.

What it would tell you is that short subject lines are good, middle-length ones are bad, and long ones are good.

And this simply isn’t the case.

Note two things: this data isn’t normalized, and you can see that we have less confidence for really short and long subject lines.

We ran the same statistics as in the blog post we mentioned at the start of this article, and got the following chart:

bad statistics email subject line length

(Correlation coefficient of -17.66%… indicating a strong inverse relationship…)

Only use short subject lines! But that would be a bad idea. Trust us. We know a thing or two about email subject lines.

Here’s bad statistics for the number of words in a subject line

bad statistics number of words in email subject line

(Correlation coefficient of -19.05%)

Here’s bad statistics for the average word length in a subject line

bad statistics word length in email subject lines

(Correlation coefficient of +5.19%)

These charts collectively tell you nothing about email subject line length. What they do tell you is that using flawed methodology can prove anything incorrectly.

Here’s good statistics for email subject line length

We took the same anonymised Phrasee user data as before and graphed the effect subject line length has on normalised open rates. It looks like this:

good statistics email subject line lengthThe solid blue line is the normalised average, once again. (Note that the Y axis is the number of standard deviations from the normalised mean open rate.)

There is a *tiny* effect. But the light-blue confidence intervals indicate this is just due to random variance in the sample set.

This proves that for any given subject line length, it is possible to have high open rates or low open rates. The impact of subject line length has a very weak, if any, impact on open rate.

Now it’s time to get gangsta… (Those damn kids on my lawn better watch out…)

Just to prove the point again… we used another methodology to make sure we were right (spoiler alert: we were.)

We ran a regression analysis to determine the relationship between open rates and subject line length

Warning: this section contains LOTS of detail. LOTS. And lots of advanced statistics. Don’t worry if you don’t understand it. We’re just ranting at this stage because we hate flawed statistical methodology so much.

Regression is a statistical method to determine how variables affect a result. In this case, we were looking to determine what effect 1) subject line length; 2) the number of words in a subject line; and 3) the average word length have on open rates.

EDIT: It’s been pointed out to us that this is a massive simplification of predictive analytics. You’re right! Our point is to simply show that *any* methodology using normalised statistics gives a better result. There are other methodologies that are perhaps more appropriate (for example General Linear Regression, as one particular troll very rudely pointed out) and if you want to have a crack, we’ll buy you dinner (KFC or BK, your choice) if the results are (robustly) different than ours.

The point remains that using normalised statistics gives you a much better indication of what works and what doesn’t. The take from this blog post is that any statistics that don’t use normalised means are methodologically dodgy and shouldn’t be trusted.

/EDIT

Short answer: barely any effect.

Next, we ran the regression using this formula:

Y = µ+ β1*X1 + β2* X2 + β3*X3

Where:

  • Y = the calculated open rate
  • µ = your average open rate
  • X1,2,3 = subject line length, # of words, and average word size, respectively
  • β1,2,3 = the calculated coefficient from our regression model

Here’s what we discovered:

Subject line length accounts for just 0.1% of email open rate variance

Here’s the full regression output:

email subject line length regression analysis

 

The “Adjusted R Square,” aka the Coefficient of Determination, indicates that this is not a good predictive model for email subject line effectiveness. 0.001393685 is low. Very low.

It can be interpreted as: “These variables don’t tell you anything about anything.”

Our not-so predictive-but-statistically-robust formula is:

Open rate = (Average open rate) + 0.018*Length + -0.139*#words + -.067*AvgWordLength

Try and plug in your own numbers. It’s an effect so small Sherlock Holmes wouldn’t be able to find it. Other factors affect open rates much more (find out more about what optimises email subject lines here.)

The only other research we could find that used good methodology, from way back in 2012, backs up our results.

So there you go.

TL;DR – email subject line length has no effect on open rates

The thesis in the original blog post that spurred our research was correct. Subject line length has pretty much no effect on open rates. But their statistical methodology was flawed. They got lucky this time.

So here’s some free advice – when you see interesting statistics about email subject lines, make sure they are statistically significant…

Because bad statistics cause bad decisions.

It’s like a jungle out there – so be careful!

PS – we use statistics like the preceding analysis to optimise email subject lines. Next step? Book a demo obviously!

Download our free “Subject Lines That Sell” report to help you become a subject line superhero.

How did you hear about Phrasee?

Sign me up to receive more of your AWESOME content.

Sign me up for your weekly newsletter.

  • Pete Austin

    Brilliant. Love it. Will link to it next week. Just one thing – this is all about correlation isn’t it? You’ve noted that there’s almost no correlation. You’ve not checked for causation.

    These two don’t always go together. You can sometimes have cause-and-effect without correlation if there are other large factors at play (For example eating more food makes you fat, and eating less food makes you thin – a cause-and-effect that’s easy to demonstrate by dieting for a week. But in the West, people are eating less food and nevertheless getting fat – there is no positive correlation between eating less and getting thinner in the overall data.)
    http://www.telegraph.co.uk/lifestyle/wellbeing/diet/10126042/Britons-getting-fatter-despite-consuming-fewer-calories.html

    You should do something like A:B testing subject lines with different lengths and comparing the results. I suspect they will be insignificant, probably with a huge variance, but that’s not yet proven.

  • Hey Pete! So, to clarify: in the sections where we say “here’s bad statistics on…” that’s all correlation. And you’re right! The correlation is the incorrect statistic to use.

    In all the “here’s good statistics” sections we’re *not* using corrrelation.

    Hey – A/B testing sounds like a great idea. Now if only there was a platform that would analyse all of these test results for you using advanced machine learning (oh wait)

    Cheers
    Parry

  • Pete Austin

    Sorry, I don’t see how the “here’s good statistics” section demonstrates causation. You charted a massive amount of existing data and convincingly showed there is no significant overall correlation, but I don’t see how it necessarily follows that there’s no causation.

    It seems possible, for example, that causation exists and some types of email benefit from having long subject lines, while other types benefit from having short subject lines. Both types of email were present in your data, so these effects happened to cancel out, and there’s no significant visible relationship in your chart.

    Please could you show your reasoning for why you’ve demonstrated a lack of causation.

  • Hey Pete!

    That’s the whole point of this post. It demonstrates a *lack* of causation. We used the “bad” statistics to show how correlation was the incorrect analysis. Then we used the “good” statistics – that is, normalised results using two robust methods – to show that there is little, if any effect of subject line length on open rates.

    If you have another analysis or methodology you’d recommend, or even better, your own statistics to disprove our thesis, we’d love to see it! But, the evidence noted above is robust, so I’d be surprised if your un-tested hypothesis is correct.

    Cheers
    Parry

  • It’s late where I am and my the math side of my brain is mush right now, so can someone please just tell me the moral of the story? What’s the ideal subject line length exactly??

    • Hey Noya! The short version is there is no such thing as an ideal subject line length. There are many other more important causal variables to focus on (which – shameless plug – is exactly what phrasee does.)

  • Pingback: Email Marketing Reading List #007 - Silverthan.com Silverthan.com()

  • Pingback: Email Marketing Statistics 2015()

  • Luke V

    Curious how this may be affected by device type. Would this still apply in mobile vs desktop? Any way to test, or are these the final, final words? ;)

    • Hey Luke – great question. We thought too that device type made a big difference. Turns out, it doesn’t. The reason? I’m not 100% sure, but my hypothesis is that 1) users multi-screen emails, so truncation points per user are hard to predict; 2) there could be an effect of pre-loading SLs with offers vs not, but I’m not sure; and 3) predicting device usage pre-send (which is when a SL is set) is difficult, so it’s an extraneous variable outside the control of SL creation.

      It is an interesting question, however I think better tests revolve around optimising the language in the subject line, not the length, and the statistics seem to back up this position!

  • Crystal C

    ….I burst out laughing once I posted this seeing my mistake. I just want to apologize now because your “Harry Palm” story stuck in my head and I actually addressed you as such and not Parry…your actual name. My bad :/

    Please still answer my question :D

    • Hey Crystal! I’m not seeing your question… can you repost it pls? (BTW Harry Palm – no worries…!!! :)

      • Crystal C

        Really? Sure I’ll repost the whole schpeel, my second comment above makes more sense in context.

        Harry! (hopefully that comes across as more “I’m enthused” and less “I’m impatiently shouting”)

        I arrived at this post via your “You’re probably doing email split testing wrong (video)” post and I have a question. First off I must say that I absolutely loathed stats in college and university and did everything possible to avoid it each time it appeared as a “must have” towards my degree. Eventually I passed the course (begrudgingly) and thought I’d be done with it but yet HERE I AM (I’m not yelling at you, I’m yelling at myself) watching your vid and reading your posts thinking “holy shit this shit makes sense and I should have learned to love stats much earlier in life” womp womp.

        Anywho, my question is; how or why can you say that using conversion rate as a success metric is a not-so-good approach because it eliminates causal factors, when on the other hand subject lines are causal to open rates?

        Now, before you or anyone else rolls your eyes (I know you are), I realize this is probably a very basic question that points to the letter grade I received once I finally passed stats. Nonetheless, your feedback is appreciated and anticipated.

        Thank you in advance!

        Cheers!

        • Hey Crystal, thanks for re-posting. Not sure what happened before. Computers are hard.

          Right – so the conversion rate as a success metric for email has two statistical limitations, as follows:

          1) There are many other random factors affecting conversions as you go deeper into a channel. For example, many people buy holidays to Dominican Republic. But if there was a hurricane there and it was in the news, no matter the subject line, guess what? Fewer conversions.

          2) Conversions are very rare events, so statistical significance is hard to come by. I’ll spare you the long, boring detail – but when you calculate “winners” of a test, using rare events are less robust than using common events. For more boring info on this, check out here: http://www.evanmiller.org/ab-testing/sample-size.html

          Here’s the thing – you need to combine stats with a bit of common sense. When’s the last time you opened, clicked, and then converted on an email? Probably it’s quite rare. But: what if you open and click on your phone, then go direct on your laptop or tablet later? This is very common behaviour, especially for high ticket purchases.

          The hard facts are: subject lines have a direct, measurable effect on open rates. That’s common sense. They also have an effect on click rates, but at a lower rate, and this is due to the inclusion of more random variance factors. SLs also have an effect on conversions, but at a further decayed rate.

          The key is to build a model that understands the effect of the SL on each of these phases, and can control based upon that. (Plug: This is what we’ve built at Phrasee.)

          That all make sense? :)

          Cheers
          Parry

          • Crystal C

            Thanks for replying Parry!

            Yes it all makes sense.

  • Pingback: Os 15 Maiores Mitos Sobre E-mail Marketing + Bônus()

  • Phillips Hayden

    Hi there,
    Well, the blog is pretty unique written in a different style. I love the illustrations that you have made in the blog (kids and lawn, lol!). Also, the statistics that have been incorporated in the blog are unique. Coming to the main point, email subject line I believe is directly proportionate to the email open rate. A good, short and crisp email subject line is a smart kung fu move to compel the subscribers to open the emails. If you kung fu move goes wrong, you might end up breaking your bones (failure in email marketing strategy). Therefore, incorporating the right type of subject line with proper word limit is the key to improve email open rate. Being quite cajoled by your blog, I am tempted to share a blog with you which you might find interesting as well.http://www.socialpositives.com/2016/07/myths-about-email-subject-lines