A/B test a campaign

An A/B test sends two to five versions of a campaign to a test group, waits, then sends the version that did best to the rest of the audience. Test subject lines, preview text, the sender name or the whole content, and pick the winner by open rate, click rate, orders or revenue.

How a test runs

1. When the campaign sends, Emailit takes the test group, a share of the recipients picked at random, and splits it evenly between the versions.
2. The campaign shows as Testing while Emailit waits for the time you set.
3. When the time is up, Emailit ranks the versions by the winner metric and sends the leading version to everyone who isn’t in the test group. The campaign is then Sent.

With a test group of 100%, every recipient gets one of the versions and there’s no winner phase.

Set up a test

1. Open the A/B test step. In the campaign wizard, go to A/B test and turn on Test versions of this email.

2. Choose how the winner is picked.

   Option	Ranks versions by
   Open rate	Unique opens per recipient.
   Click rate	Unique clicks per recipient.
   Orders per recipient	Orders credited to each version, per recipient.
   Revenue per recipient	Revenue of the orders credited to each version, per recipient.

   Orders per recipient and Revenue per recipient use Commerce, which is in alpha: only the Emailit team can choose them. A campaign that already uses one keeps it.

3. Set the test group and the wait. Test group (% of audience) is from 10 to 100 (default 20). Wait before picking (hours) is from 1 to 168 (default 4). Orders take longer than clicks, so give tests by orders or revenue a day or more.

4. Edit the versions. Each version has its own Subject, Preview text and From name. Leave a field empty to use the campaign’s value. Select Use different content to give a version its own email content; otherwise it uses the campaign content. Add version adds up to five versions.

5. Send. A/B tests send all at once, now or at a scheduled time. The other send strategies aren’t available for them.

Read the results

The campaign report shows an A/B test card with each version’s Sent, Loaded, Clicked, Orders and Revenue (orders and revenue for the Emailit team only, while Commerce is in alpha), and a trophy next to the winner. Its label tells you how confident the result is, based on the chance that the leading version really beats the runner-up:

For revenue tests, the confidence is computed on orders. The winner metric and the percentage are shown next to the label.

End a test early

While the campaign is Testing, select Send this one on any version to send it to the rest of the audience right away. This ends the test.

If every version scores the same, for example when no orders were credited yet, the first version wins.

Use the API

Pass ab_test when you create or update a draft:

{
  "ab_test": {
    "enabled": true,
    "winner_metric": "clicked",
    "test_size_percent": 20,
    "duration_hours": 6,
    "variants": [
      { "name": "A", "subject": "Our autumn collection is here" },
      { "name": "B", "subject": "{{first_name|\"Hey\"}}, new arrivals picked for you" }
    ]
  }
}

variants takes two to five versions. Each can set subject, preview_text, from_name, from_email, html and text. Send { "enabled": false } to turn the test off. Retrieve campaign statistics returns each version’s results in ab_test.

Related