Creating an A/B Test

Click the A/B test button in the builder toolbar. You then choose what you want to test:

  • Subject, preheader, or sender name – also known as an inbox test. You set the text for each variant (A through D) per field. Xendy can suggest variants per field in your brand voice using AI.
  • Content – you choose between two approaches:
    • Individual blocks: you only adjust specific blocks per variant; the rest is identical to variant A.
    • Full designs: each variant is a completely independent design.

Switching Variants in the Builder

Keep in mind that switching between variants clears the Undo history. This means you can't restore rows from a different variant.

Sending and Settings

When scheduling an A/B test, you go through three steps:

  1. Standard send details – same form as a regular newsletter.
  2. A/B settings – configure the following:
    • Test size: the percentage of your list that receives the test.
    • Wait time: how long Xendy waits before selecting the winner (choose from fixed options).
    • Winner metric: open rate or click rate.
    • Control group: optional, up to a fixed maximum percentage.
    • Mode: send the winner to the rest of the list, or a full distribution without a winner.
  3. Summary – review everything and confirm the send.

While an A/B test is running, no second send can be started.

Monitoring a Running Test

On the campaign page, during the test you can see:

  • Live preliminary stats per variant.
  • A countdown to the decision moment.
  • A graph showing how the rate has developed over time.

You can manually select the winner using the hold button, even before the wait time has elapsed. Xendy shows how certain the difference is and which variant is leading.

Viewing Results

After the test has ended, you can see:

  • Which variant won.
  • The difference in rate between the variants.
  • The statistical certainty of the result.
  • How the winner was chosen: automatically, decided early, by you, or fallback to A.

Through the new A/B test overview screen you can see all tests in one overview. You can filter by what was tested (subject, content, etc.) and see the winner, the difference, and the outcome per test.

Frequently asked questions

You can set up variants A through D, so a maximum of four variants per test.

If Xendy can't determine a winner, the outcome falls back to variant A. You'll see this reflected in the results under the way the winner was chosen.

Yes, you can manually select the winner using the hold button on the campaign page, both during and after the test.

No, while an A/B test is running, no second send can be started.

Joeri Ras

Written by

Joeri Ras

Founder & co-owner @ Xendy

Prefer to talk directly?