Testing subject lines is important. Very important. That’s how we get companies up to 417% more revenue.

But: not all ESPs are created equal!

Which ones offer the most flexible split testing capabilities? And how easy is it?

Fear not: we have all the details here!

ESPA/BMVTNo. of variantsChoose winning
criteria
Auto send winnerManually send
winner
Adestra26
Adobe Campaign
BlueHornet2
Bronto5
Campaign Monitor2
Communicator5
Constant Contact2
Dotmailer
10


Emarsys


Emma3
Experian Marketing Suite
ExpertSender5
IBM Silverpop4
Intilery
Listrak10
Lyris - acquired by Aurea4
Mailchimp3 Regular
9 Pro
MailUp
Maropost
Oracle Responsys
8
PostUp8
Pure3605
Sailthru

5


Salesforce Marketing Cloud - ExactTarget
2


SmartFocus20
Vertical Response2
Yesmail5

All info was gleaned from company websites or from company representatives. If you’re an ESP and not featured please email the correct info to awesome@phrasee.co. Also if anything is inaccurate let us know!

Adestra

A/B or multivariate testing?
Both

What variables can you test? 
Subject line, from address, reply address and content

What metrics can you use to measure deliverability success?
Open rate, click rate and click to open rate

How many variants of a campaign can you send?
26

Can you automatically send the winning campaign to the remaining data?
There is the option to manually or automatically send the winning campaign to the remaining data. You can set the amount of time you want to wait before sending the final version (30 mins up to 1 day)

Extra info

  • Option to select the percentage of the list that you want to send split test variants to
  • Test your campaign across entire list
  • Watch how customers respond to each of campaign variants, in real-time via a dashboard reporting interface
  • Scoring system to help you decide which email variant is most successful

adestra.com

Adobe Campaign

A/B or multivariate testing?
Both

What variables can you test?
Subject line and content

What metrics can you use to measure deliverability success?
Open rate, best click through, highest reactivity rate, lowest complaint rate, highest conversion rate, number of pages visited, lowest unsubscription rate and transaction amount

How many variants of a campaign can you send?
As many as you want

Can you automatically send the winning campaign to the remaining data?
You can automatically send out the winning campaign. There is also the option to manually compare the results of the campaign and decide which of the variants you will send out to the remaining data

 

adobe.com/uk/marketing-cloud/campaign-management

BlueHornet

A/B or multivariate testing?
A/B

What variables can you test?
Subject line, from name, from email, reply email, message body and send time

What metrics can you use to measure deliverability success?
Open rate and click rate

How many variants of a campaign can you send?
2

Can you automatically send the winning campaign to the remaining data?
You manually compare the metrics for each campaign to decide which is the winning variant to send out to the remaining data.

bluehornet.com

Bronto

A/B or multivariate testing?
Both

What variables can you test?
Subject line, from name, from address, date and time of send and content, but can only test one variable at a time

What metrics can you use to measure deliverability success?
Open rate, click rate, conversion rate, revenue per contact, revenue per click, click through rate or conversion/delivered rate

How many variants of a campaign can you send?
5

Can you automatically send the winning campaign to the remaining data?
Yes. You can also select the time to send the winning campaign and spread the sending of winning content emails over time e.g. over 4 hours. You cannot select and send out to the winner manually. 

Extra info

  • Champion/challenger split test option where you can run a new challenger group against the established champion (winning variant) to a small percentage of your list with no winner being sent out.
  • Option to select percentage of list to send splits to
  • Comparison report of split test results which highlights the winning criteria stat but also lists other key metrics so can compare

bronto.com

Campaign Monitor

A/B or multivariate testing?
A/B

What variables can you test?
Subject line, from name and content

What metrics can you use to measure deliverability success?
Open rate, total unique clicks or total clicks on a selected link

How many variants of a campaign can you send?
2

Can you automatically send the winning campaign to the remaining data?
Yes. You can select the number of hours after the test that you want to send the winning version out to your remaining recipients. You can also manually select the winner while test is still in progress.

Extra info

  • Option to select percentage of list that was to send each split group to
  • Able to select the length of time test runs for
  • Live test progress reporting

campaignmonitor.com

Communicator

A/B or multivariate testing?
Both

What variables can you test?
Subject line and content

What metrics can you use to measure deliverability success?
Total opens, unique opens, total clicks, total conversions and conversion value

How many variants of a campaign can you send?
5

Can you automatically send the winning campaign to the remaining data?
Yes – with manual option.

Extra info

  • Campaign management team can set up testing strategies for you
  • Able to select the length of time test runs for
  • Also offer a ‘real-time optimisation’ tool which allows users to upload any number of images and set an open quota for a test phase. The system will automatically display each variant at random and once the open quota is reached, the image that encouraged the most clicks is displayed from then on

communicatorcorp.com

Constant Contact

A/B or multivariate testing?
A/B – but has to be done manually. No built-in split testing feature

What variables can you test?
Subject line, time, content

What metrics can you use to measure deliverability success?
Reporting statistics for the open and click rates are provided and you have to manually compare these results to decide which version of the campaign to send out

How many variants of a campaign can you send?
2

Can you automatically send the winning campaign to the remaining data?
No

Extra info

  • Process isn’t automated – needs to be done manually by creating copies of the campaign

constantcontact.com

Dotmailer

A/B or multivariate testing?
Both

What variables can you test?
Subject lines, from name, from address, content

What metrics can you use to measure deliverability success?
Opens, unique clicks or ROI

How many variants of a campaign can you send?
10

Can you automatically send the winning campaign to the remaining data?
Yes – with option for manually

Extra info

  • Set percentage of contacts sending split to
  • Set no. hours to wait and then the winner will be sent immediately after this
  • Split test reporting only shows the metric that was selected as the winning criteria

dotmailer.com

Emarsys

A/B or multivariate testing?
Both

What variables can you test?
Subject line, from name, from address, content and email header

What metrics can you use to measure deliverability success?
Open rate or unique click rate

How many variants of a campaign can you send?
No limit of the versions that you can create and test, but cannot send the test to more than 90% of list

Can you automatically send the winning campaign to the remaining data?
Yes – with option for manual

Extra info

  • 1 hour minimum wait for sending version automatically
  • Each variant needs to be scheduled individually

emarsys.com

Emma

A/B or multivariate testing?
Both

What variables can you test?
Only subject lines

What metrics can you use to measure deliverability success?
Open rate is the fixed criteria for choosing winner

How many variants of a campaign can you send?
3

Can you automatically send the winning campaign to the remaining data?
Yes – with option for manually

Extra info

  • Subset that test is sent to is forced at 20% of the main data set

myemma.com

Experian Marketing Suite

A/B or multivariate testing?
Both

What variables can you test?
Subject lines, send times and content

What metrics can you use to measure deliverability success?
Opens, clicks and conversions

How many variants of a campaign can you send?
Unlimited

Can you automatically send the winning campaign to the remaining data?
Yes – with option for manual

Extra info

  • Time frame for determining results can be set
  • Can set a performance target (e.g. send to best performing only if uplift is >5%)
  • Option to test across channels (e.g. email vs SMS vs social)
  • Their capability in cells and splits is a very strong feature as it allows for multi-dimensional splits i.e. the child-grandchild style hierarchy. It also allows the user to split out through different channels e.g. one campaign could have an email cell, SMS cell and print cell etc.

experian.co.uk/marketing-services/products/cross-channel-platform.html

ExpertSender

A/B or multivariate testing?
Both

What variables can you test?
Subject line, from name, content, and delivery date/time

What metrics can you use to measure deliverability success?
Opens, clicks and conversions

How many variants of a campaign can you send?
5

Can you automatically send the winning campaign to the remaining data?
Yes – with option for manual

expertsender.com

IBM Silverpop

A/B or multivariate testing?
Both

What variables can you test?
Subject lines and from names

What metrics can you use to measure deliverability success?
Unique open rate, unique click rates, unique conversion rate and effective rate

How many variants of a campaign can you send?
4

Can you automatically send the winning campaign to the remaining data?
Yes – with option for manually

Extra info

  • Can set the time when the winner should be sent out
  • Can set percentage of data set to send test

silverpop.com

Intilery

A/B or multivariate testing?
Both

What variables can you test?
Subject lines, from names and any creative, content or timing variables

What metrics can you use to measure deliverability success?
Open rate, click rate, unsubscribe rate and the time responded by

How many variants of a campaign can you send?
No limit

Can you automatically send the winning campaign to the remaining data?
Yes – with option for manually selecting

Extra info

  • They are a omni-channel engagement technology provider so you can split test across any channel

intilery.com

Listrak

A/B or multivariate testing?
Both

What variables can you test?
Subject lines, delivery date, delivery time, from name, from address and message content – but only one can be tested at a time

What metrics can you use to measure deliverability success?
Open rate, read rate, click rate, click to open rate, conversion rate and revenue per email

How many variants of a campaign can you send?
10

Can you automatically send the winning campaign to the remaining data?
Yes

Extra info

  • Two test types: Winner Takes All whereby the best performing message is sent out to the remainder group and Standard Test whereby there is no remainder group and the split test is performed to the whole list
  • Can set percentage to send test to (defaults to 50% with Winner Takes All and 100% for Standard Test)
  • Can set test duration (hours and days) and with the Winner Takes All test you can define when the test will start – immediately or specific date and time
  • Results displayed for all metrics

listrak.com

Lyris

A/B or multivariate testing?
Both

What variables can you test?
Subject line and body content – only one at a time though

What metrics can you use to measure deliverability success?
Open rate and click rate

How many variants of a campaign can you send?
4

Can you automatically send the winning campaign to the remaining data?
Yes

Extra info

  • Can set the period of time the test runs for

aurea.com/products/email-marketing

Mailchimp

A/B or multivariate testing?
Both

What variables can you test?
Subject lines, from name, time of day or content – can do more than one at a time with MailChimp Pro

What metrics can you use to measure deliverability success?
Open rate, click rate, revenue with MailChimp Pro

How many variants of a campaign can you send?
3 and 8 for MailChimp Pro

Can you automatically send the winning campaign to the remaining data?
Yes

Extra info

  • Standard A/B testing can only test single design elements but with pro multivariate testing you can combine subject lines, from names, content and send times simultaneously to see which approach works best
  • Select percentage of recipients to send a winner or distribute the combinations across all recipients
  • Select time test should last (hours and days)
  • Multivariate reporting so that you can see the impact of each of the variables by themselves and their combination results

mailchimp.com

MailUp

A/B or multivariate testing?
Both

What variables can you test?
Subject line, call to action, the pictures, or the header

What metrics can you use to measure deliverability success?
Open rate or click rate

How many variants of a campaign can you send?
Infinite

Can you automatically send the winning campaign to the remaining data?
Yes – with option for manual

Extra info

  • Can test on up to 30% of the database

mailup.com

Maropost

A/B or multivariate testing?
Both

What variables can you test?
Subject lines, from name, preheader and content

What metrics can you use to measure deliverability success?
Top choices (selects the subject line and from name based on the highest open rate and the content based on the highest click rate), open rate, click rate and manual (select everything at later stage after viewing the reports for the test)

How many variants of a campaign can you send?
More than 2 – specific number not available

Can you automatically send the winning campaign to the remaining data?
Yes – with option for manual

Extra info

  • Report to show results for all metrics

maropost.com

Oracle Responsys

A/B or multivariate testing?
Both

What variables can you test?
Subject lines, from name and content

What metrics can you use to measure deliverability success?
Open rate, click through rate, conversion rate and unsubscribe rate

How many variants of a campaign can you send?
8

Can you automatically send the winning campaign to the remaining data?
Yes – with option for manual

Extra info

  • Select sample size
  • Select duration of test (days)
  • Reporting of all key metrics

oracle.com/marketingcloud

PostUp

A/B or multivariate testing?
Both

What variables can you test?
From name, from address, reply to name, reply to address, subject line and message content

What metrics can you use to measure deliverability success?
Clicks, opens, shares & likes, referred clicks and conversion rate

How many variants of a campaign can you send?
8

Can you automatically send the winning campaign to the remaining data?
Yes – with option for manual

Extra info

  • The evaluation period to select the best performing mailing is configurable by end users to as little as 1 hour and as much as 96 hours.

www.postup.com

Pure360

A/B or multivariate testing?
Both

What variables can you test?
Subject lines and content

What metrics can you use to measure deliverability success?
Content test defaults to clicks as winning criteria and subject lines to opens

How many variants of a campaign can you send?
5

Can you automatically send the winning campaign to the remaining data?
Yes – with option for manual

Extra info

  • Testing time can be set between 1 and 24 hours

pure360.com

Sailthru

A/B or multivariate testing?
Both

What variables can you test?
Subject lines, from name, from email, content and send time – able to test more than one at once

What metrics can you use to measure deliverability success?
Can only select winning evaluation criteria for the AB auto winner version – there are x5 options – open, click, pageview, revenue or purchase

How many variants of a campaign can you send?
AB split – up to 5 versions, AB winner – up to 4, AB Auto Winner – up to 4

Can you automatically send the winning campaign to the remaining data?
Yes – with option for manual

Extra info

  • Can select percentage of list to send each variant to
  •  Three types of test available: AB split (sent to full list no remaining send out), AB winner (winning variant selected manually), AB auto winner (winning variant selected automatically)

sailthru.com

Salesforce Marketing Cloud

A/B or multivariate testing?
A/B but…

Check out Phrasee’s SFMC app! You can MVT loads more subject lines than just A/B. Awesome!

What variables can you test?
Subject lines, from name, pre-headers, send dates and times and content

What metrics can you use to measure deliverability success?
Open rate or click through rate

How many variants of a campaign can you send?
2

Can you automatically send the winning campaign to the remaining data?
Yes

Extra info

  • Can select percentage of list to send each variation to
  • Predictive A/B test option whereby remaining sends will be sent to whichever message is best for that subscriber

marketingcloud.com

SmartFocus

A/B or multivariate testing?
Both

What variables can you test?
Subject line, send time, from email, content and dynamic content

What metrics can you use to measure deliverability success?
Clicks, opens, conversions, revenue and social sharing

How many variants of a campaign can you send?
20

Can you automatically send the winning campaign to the remaining data?
Yes

Extra info

  • Can select percentage of list to send each variation to

smartfocus.com

Vertical Response

A/B or multivariate testing?
A/B

What variables can you test?
Subject lines, from name, send date, send time and content

What metrics can you use to measure deliverability success?
Open rate, unsubscribe rate, click rate and conversion

How many variants of a campaign can you send?
2 – can split lists into more than 2 depending on the list size

Can you automatically send the winning campaign to the remaining data?
No

Extra info

  • Not an inbuilt feature – user needs to manually set up the test by creating copies of the email and splitting lists into segments
  • Straight A/B split test (50:50) with no option for winning version to be sent to remainder group

verticalresponse.com

Yesmail

A/B or multivariate testing?
Both

What variables can you test?
Subject line and content

What metrics can you use to measure deliverability success?
Open rate, click-through rate, click to open rate or purchase volume

How many variants of a campaign can you send?
4 – with option for one control variation to make 5

Can you automatically send the winning campaign to the remaining data?
Yes – with option for manual

yesmail.com

Sign up to Phrasee’s weekly newsletter. It’s awesome. We promise.

  • K.Gegeshidze

    WELL done bravooo… thanks for this contribution, very good article clearly shown comparison of all the ESPs on the market. Would be useful if you crafted article and showed list and comparison like this about deliverability service providers and showing comparison for e.g: sparkpost, mailboxlayer….
    Deliverability is the topic on what about majority of e-marketers are concerned nowadays.