AB testing the right way to generate big bucks

Reading time 5:45 mins

If AB testing is King of Conversion, Research is the Queen

AB testing provides those answers you desperately need. But to get the right answers, you need to ask the right questions. There lies the difficulty. Today I will explain how to find the right questions to ask so you don't waste your time and money with fruitless questions.

If AB testing is King of Conversion, Research is the Queen

Research and AB testing go hand in hand. Do one without the other at your own peril. Just as you shouldn't AB test without prior research you shouldn't draw research conclusions without AB testing. The sole judge is your audience. The only way to know for sure what works and what doesn't is to ask your audience. Otherwise you're just guessing. And bad guesses cost money, sometimes A LOT of money.

No research = bad questions = poor results

It's easy to jump the guns when you first hear about AB testing. You are thinking "what if I change that" or "what if I remove this". Those are the questions fuelled by your intuition. Be ware: your intuition will lead you astray almost every time. And the reason is simple: you are not your audience. What feels right to one individual (you in this case) won't apply to the whole of your audience. Worse, your point of view is biased as you are not in the shoes of a visitor or a customer.

So do yourself a big favour and discard these so-called ideas right off the bat. If you can't bring yourself to discard them completely, then open a spreadsheet, write them down and put it aside.

There is no shortcut, research is everything

I have been doing CRO consulting for many years now and I can fairly accurately predict whether or not a client of mine will get a conversion increase out of AB testing. It's simple: if we are testing ideas based off intuition I know we will be wasting precious time and money. My role then is to point out the errors made along the way, warn them and to show my expertise so they trust me to do things the right way when they realise they've been asking the wrong questions all along.

You also need to realise that proper AB testing takes time. It is not a miracle solution that will solve all your problems overnight. Taking intuition-based shortcuts or trying to rush the process will only lead (you) to poor results.

You might also be tempted to copy what a competitor is doing, thinking it's working out for them. Please don't. Even if it works for them, there is no reason it will work for you. You don't have the same audience. So if you think you can fast-track AB testing by copying a competitor, think again.

There is no shortcut. Be patient. Do your research. It will pay off big time... in due time.

Conversion research the right way

If you reached that point, you're probably thinking: "okay, okay, I need to do research, get to the point already!". Here we are then. Here is a simplified representation of the research process:


Let's take it step by step.

Step 1: identify conversion problems

The first step is to figure out what to optimise on your site. You might think you know what the biggest conversion problems on your site are, but unless it is backed up by data, it's safe to assume you're probably wrong. Don't make assumptions. Instead deep dive into your analytics data and/or review qualitative data.

Word of caution before you start: you need to be sure your analytics settings are well configured. There is nothing worse than making decisions out of inaccurate data. It will destroy any chance you have of increasing your conversion rate before you even start. So take the time to properly configure your analytics, or hire a professional to do it for you. If you don't have anyone at hand, reach out to us, we offer a Google Analytics health check & custom configuration service. Please make sure the data you're using is accurate before your proceed further / do anything else.

Understand the context

Here is what I usually start with:

  • establishing a conversion baseline over the past 12 months: what is the conversion rate, is there any seasonality, how is it evolving over time?
  • figuring out who the visitors are: what gender, age, location, device of choice, length of buying cycle, etc.?
  • understanding the site dynamics: where do visitors come from, how traffic sources evolved over time, what are the key landing pages?
  • laying out the conversion funnel: what steps are visitors required to take to convert, what % of dropouts do we have at each step?

If you are the site owner/manager you probably know all that already and can skip to the next part.

Sources of underperformance

Now that we have a clear picture of the site dynamics, let's identify sources of under-performance. This is how we will figue out what problems to solve, what questions to ask, what tests to run. It is all about segmenting your data and comparing performance, both in terms of engagement and conversion. Here is an example of metrics to compare segments for an eCommerce site:

Segments Sessions Bounce rate Time on site Pages/session Conversion rate Avg order value Revenue per visitor
Segment 1 right-aligned $1600
Segment 2 centered $12
Segment 3 are neat $1
Total are neat $1

Looking at this data here is what I would conclude: Segment 2 is clearly under-performing...

You should do the same with all major Google Analytics reports, trying to figure out what segments are underperforming. You will end up with a list of what I call "Key segments". These segments are telling you where you are losing money. You don't know why yet, but you know where to look. Here are segments that would make for a good start to review

Traditional Google Analytics segments:

  • desktop vs. tablet vs. mobile
  • new visitors vs. returning visitors
  • per traffic source
  • per landing page (bounce rate and conversion rate)
  • per location (country / city), per language
  • per number of sessions
  • sessions with conversions vs. sessions without conversions
  • and many more, complete that list based on what you find with the segments above

If you have set up advanced tools on your site you may be able to go even deeper, with segments based on visitor behaviour.

Advanced behavioural segments:

  • conversion rate for visitors using search vs. visitors using navigation
  • true bounce rate per segment (not 1-page visits but visits lasting only a few seconds)
  • per time of day, per day of the week
  • based on weather or temperature
  • based on purchase history
  • and many more depending on your industry and conversion goals

Compare with qualitative data

Step 2: list as many potential solutions as possible

What are the chances your first idea is the best one?

Step 3: prioritise your ideas


Go to

Share your thoughts with the community

Blog Comments powered by Disqus.

Growth Bootcamp