Optimizing your website is key for the overall performance of your platform. But what do you need to optimize and how do you know the optimization will work out the way you want it to? With A/B testing you can figure this out by trying out what works best for your platform and your visitors.  

In this blog we will guide you through all the steps you need to take to perform a nice, clear and practical A/B test. In the end you can implement the best of the two variants on your website and you are one step closer to the ideal website for the future.  

Step 1. Identify optimization possibilities

It all starts with a hypothesis or research. You need to identify a problem or set a goal before A/B testing. You do this by analysing data, such as the behaviour of real users on your website. These insights help you addressing a real problem. You can use different data sources to find optimization possibilities, such as uMarketingSuite, Website Analytics, dashboards, usability research and external expert reviews.  

There are probably a lot of optimization possibilities based on your research, some of them are good and some mediocre. You can't test everything all at once and you have a limited amount of traffic. That's why it's extra important to prioritize. But where do you start? To determine which A/B test has priority, you can use the prioritization model PIE. PIE stands for Potential, Importance and Ease. The average of these three factors determines the priority and the A/B test you start on first.  

For example:

Step 2. You need to have significant scale to A/B test efficiently

To make sure the outcome of the test is not by chance, you can use a statistical significance score to ensure the validity of the result. You need to be at least 95% sure if a certain variable actually works better. To reach this, you will need a lot of traffic or a large deviation in conversion, minimal detectable effect (MDE). Fortunately, uMarketingSuite has automated all of the math for you.

A lot of marketers make the mistake by running the A/B test as long as it is needed to get the right amount of traffic. But in this way you won’t get good results. So before you start with A/B testing, calculate if you have enough traffic to have significant scale to test.

To determine whether your website is eligible for A/B testing, you can use the ROAR model.

This model consists of 4 phases in the field of testing :

• The risk phase: less than 1000 conversions per month
• The optimization phase: more than 1000 conversions per month
• The automation phase: more than 10,000 conversions per month
• The rethink phase: in this phase so many tests are completed that the website is constantly being improved.

The more conversions per month, the more reliable the A/B test. Is your website in the Risk Phase? In this phase you can A/B test, but it is important that you take into account that erroneous conclusions (type 1 error) can be drawn. As a result, a hypothesis (H0) can be wrongly rejected. The advice in this phase is to test major adjustments. Because small changes such as micro copy, are less likely to have a large and noticeable impact on conversion rate.

Step 3. Choose the variable you want to test 

So now you have determined optimization possibilities and you have prioritized. When you think you have enough traffic to validate the A/B test, you can start with picking variables to test. You probably want to test numerous variables, but to measure the impact of the change, it’s best to test one variable at a time.

Step 4. Make a hypothesis  

A hypothesis is an assumption about how you intend to solve a conversion problem or optimize your platform.
• This should solve the conversion problem .
• This should provide marketing insights, so that you can also test these findings for other parts on the site.
• You must be able to determine whether the test was successful or not.

There are different ways to set up an A/B test with a substantiated hypothesis. You can have a remote usability test performed or you can use customer feedback. Also, you can analyse the webpage by using a conversion optimization model. Such as the LIFT model. LIFT stands for Landing Page Influence Functions for Tests.

This model uses the following conversion factors to construct a test hypothesis:
• Value proposition: what are the benefits of your product/service for the user?
• Relevance: how relevant is your content to the user?
• Clarity: do you communicate one message through your content?
• Distractions: what content distracts from the ultimate goal?
• Uncertainties: which content can make the user uncertain?
• Urgency: do you communicate through your call-to-actions that the user should take action now?

Based on these insights you can phrase your hypothesis:
Changing (the element being tested) from ___________ to ___________ will increase/decrease (the defined measurement).

Keep in mind that it is important that the impact of the change is measurable in quantifiable terms (contact forms, conversion rate, exit rate etc.). Choose a primary metric to focus on. This will help you examine the results.
You can now think about the solution you want to implement.

For example:
“Shortening the form by deleting optional fields such as organisation and telephone number will increase the number of contacts collected.”

Step 5. Create a control (A) group and a challenger (B) 

After you have established a number of hypotheses, it is time to work things out into a new design for the relevant elements that relate to your hypotheses. For example, if you think that shortening the form will increase the number of contacts collected, you want to test a standard contact form (A) versus a shorter contact form (B). You will have to create a shorter contact form, in this case without the fields organisation and telephone number.

Step 6: Split traffic to variations A and B 

In uMarketingSuite (the marketing toolbox for Umbraco) you can easy setup the A/B test; you can determine the sample size, the duration and the winning metric (contacts collected). The duration of the test depends on how much data you have collected. The more traffic you get, the more reliable the test becomes. This way you make sure that the outcome of your test is statistically significant. uMarketingSuite will estimate how many days you will likely need to get accurate and reliable data. This will depend on several different factors. This way you can start your test and lean back.

Step 7. Measure the results 

Analyse the results of the A/B test and determine the winner. Focus on your goal metric and find out whether the tested variable has had an impact or not.

Step 8. Implement outcome on website 

Is your hypothesis accepted? Then it is time to implement the outcome on your website. You might consider to use the lessons you learn in future experiments. Just remember that an A/B test is case-specific and that it does not guarantee success in the future.

Is your hypothesis rejected? You can choose to leave it this way or you can run another test. It's up to you! The important thing are the lessons you learned.

In uMarketingSuite, when you start a new test, you will be provided with an overview of all previous tests for the page. This way, you can gather inspiration for your new test, learn lessons from the past and make full use of all of your previous experience.

In short, there are different ways in A/B testing, the main thing you need to keep in mind when you start with A/B testing, is that it's important to do research and to make substantiated hypotheses. This way you are doing A/B tests that make impact.

Do you need advice in setting up your A/B test?
Our CRO-specialists would gladly help you in getting the most out of your website.
Book a demo to see the uMarketingSuite in action