“Open Rates are Dead” And Other Lies People Tell You About Apple’s iOS15 Updates

Open rates aren’t dead, but they are changing. Here’s what you need to know … according to actual statistics.

When the news of the iOS15 update came out, almost immediately self-proclaimed experts around the world sounded the death knell for open rates. “The open rate is dead,” they’d proclaim. “Let’s get rid of vanity metrics,” they’d decry. “Hurr da durr da durr,” they’d grunt.

Here’s the thing: open rates aren’t dead, they aren’t dying, and they won’t be anytime soon. What is happening is that they are changing, and these changes will affect you.

Here’s what you need to know…

Open rates are getting noisier, not dying

With these iOS15 updates on email tracking, we know that open rates will be getting noisier. But the thing is, everyone’s open rates will be getting noisier.

In essence, the proportion of your audience showing opens is going to be artificially inflated by somewhere between 10%-50%. To put this in context: if you previously got a 10% open rate, as the iOS15 updates get adopted you’ll find your open rates will start to look more like 19%-55%.

Yes, that’s a big range – and if you benchmark success based upon a time-series of campaigns and their open rates, you may want to consider another approach.

But if what you care about is optimizing down-funnel performance… then your average open rate being 10%, 19% or 55% doesn’t matter.

Why?

For optimization purposes, you care about relative open rates, not absolute. Let’s say you’re running an A/B test, and your objective is to find which variant is optimal. Or an A/B/C test. Or an A/B/C/…/Z test. It doesn’t matter. The same principles apply.

For optimization purposes, what you care about is which variant is optimal compared to the alternatives – not what any individual open rate is.

To put this in context – let’s say your average open rate of 10% becomes 30% after these iOS15 updates, and your results look like this:

The best variant here is Variant A , with a 20% relative uplift versus the average. That’s what you care about.

Remember: your goal here is to make the optimal decision. On an individual campaign basis, optimality is about relativity, not absolutism.

In essence, inflated open rates don’t matter when you’re focused on optimization. It just means your baseline is different.

(Note: this assumes iOS15 users are randomly distributed in your test samples. Your ESP *should* be doing this already, but it never hurts to check with them!)

Relying solely upon clicks and conversions may lead to bad decision making

Another mistruth experts have been spewing is how you now need to focus on clicks, or on conversions.

Well, it’s sort of a mistruth.

I agree with them 100% that ultimately you should care about clicks and conversions. And all optimization systems – including Phrasee – should ultimately drive towards maximizing both clicks and conversions.

But, most of the time, you can only do that by including opens in your optimization models.

See, all of these events are known as binary variables – people either open or don’t, click or don’t, convert or don’t. When you get enough samples, binary variables converge on a normal distribution (further reading: Central Limit Theorem).

Let me put it in non-nerdy terms. A binary variable, with enough samples, will start to look like a bell curve. But it takes a LOT of samples.

Let’s say your list is 1 million people, and you have a 10% open rate, 1% click rate, and 0.1% conversion rate.

That means you have 100,000 opens to base your decision on. Nice!

But you’ve only got 10,000 clicks. Not quite as nice.

And only 1,000 conversions. That’s… not great.

So here’s what you need to know:

To make sound optimization decisions that are based upon robust statistical modelling, you need lots of samples. Unless you have HUGE lists, clicks and conversions should not be used in isolation as your optimization variable. If you do this, your decisions, in the long run, will be poor.

Considering multiple metrics is better than any individual metric

Of course, if you have 100,000 opens, 10,000 clicks, and 1000 conversions, there is obviously click and conversion information that will be missed in the open data. And this information matters.

But, as discussed in the previous section, it is suboptimal to consider clicks or conversions in isolation… so what are you supposed to do?

Simple: use an “ensemble model”.

An ensemble model is when you fuse different information sources, each with their own strengths and weaknesses. When you combine these diverse data points, the accuracy of the resulting metric, in the long run, is higher than any individual metric on its own.

This holds true across myriad applications and domains. One well-known example, which you probably use all the time, is Rotten Tomatoes, the movie rating site. Their “Tomato Meter” fuses reviews from a number of different critics. Some critics like a movie, some don’t… and individually each review has a high likelihood of being wrong. But what the site does is take multiple points of low-accuracy information… and fuse them into a single score which is highly accurate. You then use this score to (hopefully) make optimal movie selection decisions.

You probably use Rotten Tomatoes or sites like it all the time. And it makes sense – don’t take one person’s word for it. But if loads of people are saying the same thing, there may be something to it, and you can use that information to make optimal decisions.

With email, it’s no different.

Opens are a useful metric – but with these iOS15 updates, they’re becoming noisier. So they’re still useful, just not quite as useful as before.

Clicks and conversions are also useful, but as rare events, it is risky – from a statistical optimization standpoint – to rely upon them alone.

But if you were to create a “Tomato Meter” for email optimization – that is, a score that fused numerous sources of information to result in better optimization decisions than before – then that sounds like a winner.

That’s why we built Phrasee Score

Phrasee Score does exactly what I’ve just described – it fuses numerous information sources, including opens, clicks, conversions, unsubs and our deep-learning-predicted priors, to make better long-run optimization decisions.

TL;DR: opens are still cool, they’re not dead, and you should still use them.

Opens are still a very useful metric, although they’re noisier than before. You can still make sound optimization decisions based upon opens alone, and unless your list is mega big, your decisions will probably be better than relying upon down-funnel metrics like clicks or conversions in isolation.

Another option is to use advanced information fusion techniques to increase the accuracy of your optimization decisions. Phrasee’s version of this is Phrasee Score.

The problem ain’t the problem. Coping with the problem is the problem. Apple’s iOS15 updates are a spanner in the works. But they’re here, and they’re not going away. Some have complained about it, some have made misinformed statements about what it means, and some have remained silent.

Here at Phrasee, we’ve done the opposite, and we encourage you to do the same:

• Embrace the change, instead of fighting it in futility
• Understand the statistics and what they actually mean
• Move forward with a solution

Things have changed. And they will change again. And again, and again, and again. And when they do change, I urge you to take a page from Phrasee’s book and embrace the change. Because fighting it will do nothing but make you very, very tired.