What is A/B testing?

A/B testing, also known as split testing, is one of the easiest and most efficient ways to improve the emails you send to your customers. It involves testing two different versions of an attribute at the same time to see which is more successful; for example, sending half your customers an email with a blue call to action button, and other half an email with a green call to action button.

It may seem like only a small change, but it’s surprising how much difference it can make to your leads and sales, particularly as a lot of small changes quickly add up!

You can keep testing and retesting every single aspect of your emails – text, images, length, interactive elements, and everything in between – to optimise your marketing messages for maximum success.

---

How do you measure success?

There are three key metrics for seeing how successful your emails are with your subscribers:

• Open rate – how many people open your email.
• Click rate – how many click a link in your email.
• Goal completion – how many successfully complete an aim you set, e.g. download a resource or make a purchase.

Open rate is a good indicator of how your sender name/email address comes across (e.g. getting an email from ‘Sales at Blue Widgets’ vs. ‘Karen at Blue Widgets’), but more significantly helps you test the success of your subject lines. A low open rate tends to indicate that your recipients aren’t interested enough in your subject line to click through and read the rest of your email. This is where split testing comes in, allowing you to test different lines of text with your customer base to see what kind of message grabs their attention. Check out our blog post Crafting the perfect email subject line for some tips to get started.

If you have a good open rate, your next port of call is the click rate, or click-through rate (CTR). This is a much broader area as there are so many factors to take into consideration. Are you offering your readers something interesting? Do they prefer long or short emails? Are text links or buttons more successful? How do they react to images? Is a humorous or serious tone more appealing? There are hundreds of different aspects you can test and change in order to create emails that work for your audience and increase your success.

This ties in nicely with directing your recipients towards completing a goal. The goal you set will depend on your website and business aims; the most common is directing traffic to your website in order to make a purchase or contact you (lead generation). A goal may consist of multiple actions to reach an end point, for example enticing your readers to generate a unique discount code and then using it to place an order on your website. In order to increase goal completion, you need a solid foundation of a good open rate and good click-through rate, but also reasonable expectations on your part as well as clarity and simplicity for the reader so they understand what they’re expected to do and can do it in the easiest possible way.

---

How to start testing your email campaigns

Once you’ve set up and created your email campaigns, you can choose which you want to send out and to whom. For example, you could send one email to 50% of your subscribers and another email to the other 50% to test two different subject lines. Since you can test up to five campaigns at once if you like, you could potentially send Email A to 200 people, Email B to 500 people, Email C to 700 people and so on (although we recommend only testing two variants when you’re starting out to make the results easier to interpret). You don’t have to send an email to all your subscribers either; you could test Email A with 10% of people, Email B with another 10% of people, then send the ‘winning’ email to the other 80%, or not at all.

You can choose the criteria to determine the ‘winning’ email, depending on whether you’re focusing on open rate, click-through, or goal completion, and your test can run from an hour all the way up to a week. It’s a balance between giving your recipients enough time to open your email and being realistic about them ever opening it at all. The amount of time your recipients typically take to open your emails will largely depend on their lifestyles and the time of day you send them, but don’t forget to allow for seasonal fluctuations as well. For example, people may be quicker at opening emails in the run-up to Christmas because they’re looking out for offers, but they may abandon their account for several days over the actual holiday period…your tests will tell you how your customers typically behave! 

Once you’ve completed your split test, you can use the results to determine what to change and test next time to build on your success. Then repeat for your next campaign. Over time your emails become more successful, resulting in more sales and profits for your business. You also learn more about your customers’ behaviour and preferences, which can be helpful when applied or tested in other areas, such as your website pages or advertising campaigns.

---

Further reading

Which Test Won – a great source of inspiration and results of other people’s split testing.

State Your Hypothesis: A Scientific Approach to A/B Testing – the example uses a web page, but the ideas can easily be applied to emails.

Using data science with A/B tests: Bayesian analysis – a technical/mathematical article for working with small amounts of data

 

(Header image credit)