Good A/B Testing Practices

UX DesignChristian Vasile • July 13, 2013 • 7 minutes READ

However, A/B testing practices are still not so popular out there and are generally used by professionals only. And this is probably because most people are not aware of this technique. This article is here to teach you more about it and to guide you towards good A/B testing practices.

What is A/B testing?

A/B testing is exactly what it sounds like: there are two versions of an element (which in our case is an interface or a website) and a metric which will define the success for each one of them. This is used in order to decide which version is better out of the two available and this can be done using different tools.

Let’s say you have these two designs and you don’t know which one is better for you. What you do is you equally split the incoming traffic on your site towards one of these versions and after a pre-defined period of time you start measuring the metrics. This practice can give you an insight into many things with conversion rate, call-to-action performance and bounce rate among others. Out of the two versions, in the end you will chose the best one, based on the metrics you are interested in.

Where to start?

This depends most of the time on the goals you set for your website. It might be a website that needs to sell. It might be a landing page with a sign-up form. It might be a contact form. It can be anything where user behavior is important to your goal. You can test many things by using the A/B technique, but the ones that are tested more often than others are:

• Images placement
• Pricing showcasing
• Sign-up form
• The call-to-action element
• Headline descriptions

Now that you have a basic understanding of what A/B testing is, let’s take a look at some of the things I encourage you to do and at some of the things I don’t recommend, as the latter are most likely going to influence your results negatively.

What to do?

They say patience is a virtue, all of us have heard this one at least once before. Not giving up after a short period of time is crucial in A/B testing, because waiting a bit longer might help you get important results. You can run A/B from a few days up to several months, it all depends on some variables which you can calculate here, in case increasing the conversion rate is what you are looking for.

A/B testing tools usually have a feature that should remember where users have been on your page the last time they visited, and only redirect them to that one. Showing users a different variation every time he enters your page will most likely confuse him and not help you too much with your testing either.

A/B testing needs to be consistent on your whole webpage. Again, as in the tip above, if you test an element that is visible on multiple pages, make sure each user always gets the same variation. Showing a user different iterations when he browses through your website will, again, confuse him and help you less.

I recommend you to test as much as you can. It is possible that on your first test, maybe even the second one, the results will be neutral or even negative. The third time they might turn out to be positive, so do lots of A/B testing. Doing A/B is actually quite cheap and doesn’t require too much effort on your side. As long as you have an iteration of your current website, regardless of how small it is, test it! You might be surprised of how much small details can affect your conversion rates.

Another thing that I recommend is continuing doing A/B regardless of how positive and helpful the results were last time. Adding positive results over some other positive results will only help you improve achieve your goal faster. Let’s make a hypothesis. We have this start-up that wants to increase their conversion rate, currently at 23%. They do A/B for two months and increase the rate to 29%. Many will choose to stop here, thinking that 6% (more than 25% increase of the initial rate) is quite a decent result. My advice is to continue. Unless you have 100% conversion rate (which you’ll never have, by the way) you can continue. A/B is so cheap to do that it is almost stupid not to take it into consideration.

You always need to test variations at the same time. Testing variations a week after another, for example, is the wrong way of doing A/B. Traffic needs to be split between different variations at the same time, because it might happen that in a week you have bad sales and in the next week you have good sales. This doesn’t necessarily mean the second variation is better. It might mean the second variation was tested right after people got their wages. Or for whatever other reason you might not be aware of. Always split test in the same time!

What NOT to do

Don’t include old visitors into new tests if your variations are very different than the current website (also called the control). You don’t want to shock visitors who are used to your site with such a sudden redesign.

Never let gut feelings decide over the test results. You might not like the big blue button, but if tests show it is better, then go with it. This is not always about you. Actually, it is never about you, keep this in mind. A/B testing is never about what you like, but about what works. Don’t reject working results just because you are biased towards another variation.

Don’t only test design and aesthetic issues. Keep in mind that content is equally important. This can mean body text, but also paragraph headlines and website headlines. These might seem unimportant, but they can offer a new visitor a wrong perspective over what is happening on the site. The right headline will not help you sell more, but it might help you decrease your bounce rate.

If you have several iterations in mind, don’t only test two at a time. You can test multiple iterations, but the more variations you have, the more daily visitors you need. You can’t split 100 visitors to 6 variations, it is simply not going to be effective – you will need to run the test for a longer period of time, which might not be cost effective after all.

Let’s say we have this website with a conversion rate of 30% which needs to be increased to 40%. The website has 10.000 visitors per day. A test with 6 variations would probably need to be run for a few days in order to get proper results. However, if the website only has 200 visitors per day, it would take almost four months for the same test to give the same results.

A/B testing is something you could already start with and there are many tools for it on the web. Therefore we gathered some of the best ones to make it easier for you.

Optimizely

Optimizely is one of the most professional A/B tools on the web. It might cost a bit more than some other ones (you get a free month trial though), but they offer some great features. This is definitely the best place to start if you are serious about A/B testing and want to jump into it straight away.

Visual Website Optimizer

We already showed you their great free calculator for A/B testing, but they are much more than that. Visual Website Optimizer is another great tool with some good pricing plans and even a free trial with limited features. Definitely one of the choices for anybody who wants to try split testing.

Unbounce

Unbounce is great for testing landing pages and it is not only a tool for testing, but it also allows you to create landing pages without you having to code them yourself. Starting testing is very quick. And they also have a nice video on the home page which explains their benefits.

There are some other tools that might be good for you as well, but the ones above should be enough for you right now. Google also had some A/B tools before, but they implemented them in Analytics, so if you are using Google Analytics, you could do split testing with that as well.

A/B split testing is one of the most powerful techniques we can use nowadays to “talk” to our users without being face to face with them. When you don’t necessarily have the time to go out there and test iterations (you should have time for it though!), A/B is a great way of doing it from in front of your screen.

Remember that you can always improve something on your website, regardless of what kind of a website you have. Working with small variations and implementing A/B split testing using them will, without doubt, help you achieve better results with your website or application.