# A/B test a campaign

> Send up to five versions of a campaign to part of the audience, pick the winner by opens, clicks, orders or revenue, and send it to everyone else.

An A/B test sends two to five versions of a campaign to a test group, waits, then sends the version that did best to the rest of the audience. Test subject lines, preview text, the sender name or the whole content, and pick the winner by open rate, click rate, orders or revenue.

## How a test runs

1. When the campaign sends, Emailit takes the **test group**, a share of the recipients picked at random, and splits it evenly between the versions.
2. The campaign shows as **Testing** while Emailit waits for the time you set.
3. When the time is up, Emailit ranks the versions by the winner metric and sends the leading version to everyone who isn't in the test group. The campaign is then **Sent**.

With a test group of 100%, every recipient gets one of the versions and there's no winner phase.

## Set up a test

1. **Open the A/B test step.** In the campaign wizard, go to **A/B test** and turn on **Test versions of this email**.

2. **Choose how the winner is picked.**

   | Option | Ranks versions by |
   | --- | --- |
   | **Open rate** | Unique opens per recipient. |
   | **Click rate** | Unique clicks per recipient. |
   | **Orders per recipient** | [Orders credited](/docs/commerce/#how-orders-are-credited-to-emails) to each version, per recipient. |
   | **Revenue per recipient** | Revenue of the orders credited to each version, per recipient. |

   **Orders per recipient** and **Revenue per recipient** use [Commerce](/docs/commerce/#who-can-use-commerce), which is in alpha: only the Emailit team can choose them. A campaign that already uses one keeps it.

3. **Set the test group and the wait.** **Test group (% of audience)** is from 10 to 100 (default 20). **Wait before picking (hours)** is from 1 to 168 (default 4). Orders take longer than clicks, so give tests by orders or revenue a day or more.

4. **Edit the versions.** Each version has its own **Subject**, **Preview text** and **From name**. Leave a field empty to use the campaign's value. Select **Use different content** to give a version its own email content; otherwise it uses the campaign content. **Add version** adds up to five versions.

5. **Send.** A/B tests send all at once, now or at a scheduled time. The other [send strategies](/docs/campaigns/sending-options/#send-strategies) aren't available for them.

> **Opens are a weak signal:** Apple Mail loads every email's images, which counts as an open whether or not the person read it. Click rate, orders or revenue usually pick a more reliable winner.

## Read the results

The campaign report shows an **A/B test** card with each version's **Sent**, **Loaded**, **Clicked**, **Orders** and **Revenue** (orders and revenue for the Emailit team only, while Commerce is in alpha), and a trophy next to the winner. Its label tells you how confident the result is, based on the chance that the leading version really beats the runner-up:

| Label | Meaning |
| --- | --- |
| **Statistically significant** | 90% or more, with at least 50 recipients per version. |
| **Promising** | 75% to 89%, with at least 50 recipients per version. |
| **No real difference** | Under 75% even though each version reached at least 1,800 recipients. |
| **Too early to tell** | Not enough data yet. |

For revenue tests, the confidence is computed on orders. The winner metric and the percentage are shown next to the label.

### End a test early

While the campaign is **Testing**, select **Send this one** on any version to send it to the rest of the audience right away. This ends the test.

If every version scores the same, for example when no orders were credited yet, the first version wins.

## Use the API

Pass `ab_test` when you [create](/docs/api-reference/campaigns/create/) or [update](/docs/api-reference/campaigns/update/) a draft:

```json
{
  "ab_test": {
    "enabled": true,
    "winner_metric": "clicked",
    "test_size_percent": 20,
    "duration_hours": 6,
    "variants": [
      { "name": "A", "subject": "Our autumn collection is here" },
      { "name": "B", "subject": "{{first_name|\"Hey\"}}, new arrivals picked for you" }
    ]
  }
}
```

`variants` takes two to five versions. Each can set `subject`, `preview_text`, `from_name`, `from_email`, `html` and `text`. Send `{ "enabled": false }` to turn the test off. [Retrieve campaign statistics](/docs/api-reference/campaigns/stats/) returns each version's results in `ab_test`.

## Related

  - [Sending options](/docs/campaigns/sending-options/): Smart sending, UTM parameters and send strategies.
  - [Campaign reports](/docs/campaigns/reports/): Every metric, tab and export.

---
Source: https://emailit.com/docs/campaigns/ab-testing/