Variant testing is one of the best ways to find out what works for your visitors. Whether testing the location of a button, a redesigned menu, or an entirely different website, variant testing gives you a chance to have hard data behind your design decisions.

What is variant testing?

Variant testing comes in two forms – A/B testing and multivariate testing. Both of them have their use in design, whether creating a new website, testing out an email, or running a print campaign. With A/B testing, you are comparing two variations of a single element, and with multivariate testing, you are comparing variations of multiple elements.

A/B Testing

A/B testing (or split testing) is when you test two variations of a single item, such as changing a button from blue to red, moving images, or altering the text. You present both versions at the same time, split between your audience, and then judge the response you get based on the goals you set.

Examples of A/B testing:

Joan runs a bookstore. Her monthly newsletters are very popular, but don’t result in the sales conversions she was hoping for. She can see if changing the style of the call-to-action buttons in her newsletter will result in more conversions.

Dante’s web hosting company is really taking off, but he’s finding that many of his customers complain about how they’ve ordered the wrong product. He can see if changing the product descriptions cuts down on the number of complaints.

Emily has a programming blog that seems to be generating a lot of traffic, but it looks like people aren’t engaging with her online courses due to the other advertising she has on her site. She can see if moving the advertising into a new location helps her customers find her courses.

With each, they create two versions of the page, changing only one thing on it. They put both versions live at the same time, and wait to see what the data will tell them.

A/B testing is a great way to test single things that may have big results. If you want to see if one thing is currently causing you issues, A/B testing gives you a way to be certain.

Multivariate Testing

Multivariate testing is when you test two variations of multiple items, spreading out the variations exponentially. While A/B testing produces two versions of a page (each with one different thing on them), multivariate testing can produce an immense number of pages, all depending on how many items on a page you want to test.

Examples of multivariate testing:

Lin’s photography website is bringing in customers, but people have remarked that it can be difficult to navigate. By trying out a new menu, a new size for her images, and a colour scheme change, she’s hoping to make it easier for her customers.

Andrew’s new organisational app is taking off, but he’s wondering what might be the best way to get customers to upgrade to the paid version. Trying out different locations for his advertisements will give him a better idea about conversions.

Penny’s company potentially has a new logo. While it works well with the existing colour scheme, she wants to try it (and the current logo) out with a few other colour schemes as well to see if things could be better.

With each, they create two versions of each element they want to test, and then create the pages needed to test out every version of the element against each other.

So, for example, Lin’s website has three elements she wants to test (Menu, Size, Colour). She creates two versions of each element (Menu A, Menu B, Size A, Size B, Colour A, Colour B) and then creates pages that test all the versions working together:

Menu A + Size A + Colour A
Menu B + Size A + Colour A
Menu A + Size B + Colour A
Menu A + Size A + Colour B
Menu B + Size B + Colour A
Menu B + Size B + Colour B
Menu B + Size A + Colour B
Menu A + Size B + Colour B

With only three elements being tested, there are only 2³ variations. As the number of elements increase, the variations increase exponentially, and you could end up with more variations than you have visitors.

When to use which method

A/B testing is ideal for when you’re making dramatic changes and want results quickly. It will be easier to tell which version is more effective faster, as the data will only be split into two sections. It’s also good if you are testing somthing that doesn’t get a lot of traffic, as you’ll get plenty of data without needing plenty of visitors.

Multivariate is good for subtle changes and when you want to understand how the elements on your page work together. You do need a lot of traffic in order to get meaningful results, however, as you are dividing up your audience into small bunches. But if you do have the traffic, the multivariate testing can give you good data that you can then take forward into larger projects, such as complete website redesigns.

Choosing the right variant testing software

Once you’ve decided which test you’re going to implement, there are many software companies out there willing to work with out to deliver those variations to your visitors.

Google Analytics has Content Experiments built in, giving you a slightly different version of variant testing, where you can test up to 10 full versions of a single page. This is convenient for A/B testing and most multivariate testing, but does limit you if you were planning on testing the same element across several pages. It’s free to use as part of your Google Analytics account, but any visitors who have installed the Analytics opt-out browser add-on for privacy purposes will not be a part of your data set.

Monetate is subscription-based software that gives you a convenient way to test not only websites, but also mobile apps and emails. This makes it a good option if you plan to do a lot of testing across different platforms, but might be overkill if you only need to test one thing.

VWO (Visual Website Optimizer) is purely for A/B testing, but is affordable and gives you a wide range of options for your site and your data. Popular and easy to use, it isn’t for expert variant testers, but can be a great starting point for beginners.

Optimizely works for websites and mobile apps, and is one of the most popular out there. They also have a wide range of e-books, blog posts, and white papers that provide advice and information on variant testing.

Things to remember when testing

Don’t stop the test too early

By finishing the test early, you could be reading too much into the small amount of results that you have. Just because 5 people spent more time on Version A than Version B does not meant that Version A is better – especially if only 20 people came to your site. While most variant testing packages come with statistics packages that explain the data to you, you can also use VWO’s Split Test Significance Calculator as a good indicator of where your data is.

Show repeat visitors the same version

Don’t shock your regular customers with new versions every time they visit. Most tools will have cookies to make certain that every visitor sees the same thing, but check before you start. And if you have a way to have your current customers stick with the old version and only show the new version to your new customers, go with that – the last thing you need during the testing period is having to answer support queries as well.

Don’t let your instincts overrule the data

While you might be absolutely convinced that your website needs a particular colour scheme or that the text in the button should be one way, the data tells you the truth. It doesn’t matter what you think is better – it’s what converts that matters. And if you didn’t want more conversions, why are you doing the testing in the first place?

Swift Icons by Darius Dan for Smashing Magazine.