Steps to take before A/B testing

Stop. Do This Before You A/B Test

Here’s one of the most common misconception of conversion rate optimization: A/B testing will solve all your problems. I’m sure you’ve either read or heard that before – often given as advice when someone complains their conversion rate is low: “Just try A/B testing the color of your buttons” or “I heard orange buttons convert better, test that!”

I mean, I’m sure you’ve either read or heard that before – often given as advice when someone complains their conversion rate is low: “Just try A/B testing the color of your buttons” or “I heard orange buttons convert better, test that!”

I see it every day, and it makes me cringe. So before doing any further testing, read this, it’s for your own good.

Why A/B testing alone won’t solve your conversion rate problems

A/B testing is fantastic, and it’s safe to say that it’s the most widely used method for validation of conversion hypotheses for websites. Done correctly, a successful test, followed by the implementation of the winning variation is how you can continuously improve your website; more signups, increase in revenue, or whatever your goal is, it might only be a test away.

Most people understand the above, but very few know how to properly execute a test, and let alone how to plan for it. And that’s where it gets problematic.

Conversion rate optimization is a process with multiple steps:

1. You have to define your business objectives

2. Find out how those are connected to your website, and identify your key metrics and KPIs

3. Using analytics, you have to analyze your data and pinpoint the 20% of that produces 80% of your results. (e.g.: the few pages on your website that most visitors visit or/and convert)

4. You dig deeper into potential conversion barriers and mine for more information using surveys, user testing, click maps, heat maps, scroll maps, and so on.After a potential conversion barrier is acknowledged, you formulate a hypothesis stating why the identified element should be improved.

5. After a potential conversion barrier is acknowledged, you formulate a hypothesis stating why the identified element should be improved.

6. An A/B (or Multivariate) test is created to either validate or invalidate your latest hypothesis.

7.When the test is concluded, if your variation won, it gets implemented, and if the base copy wins, an iteration of your hypothesis has to be created. 

Did you see it? A/B testing is one of the last steps in the optimization process. Many people will jump right to it without being aware other steps exists, and this will do nothing but produce mediocre results.


Why skipping steps will hurt your optimization efforts

The process above does not exist to make us work harder, nor to make us lose time. The truth is that I use this process and always recommend to follow it religiously because it has the power to make our optimization efforts much more worthwhile.

Let’s break the process down step-by-step and I’ll explain the importance of each.


Discovery & Planning

  1. Defining your business objectives

Defining business objectives is the first thing I do when I start optimizing a new website. The reason why it is so important is because it’s easy to lose track of what our website really needs to accomplish on our behalf. I’ll talk more about website goals below, but in order to be efficient at pinpointing what our exact primary and secondary website goals should be, the business objectives must be clearly defined.

Business objectives should be simple, clear, and easy to measure. Once you got it, it will be much simpler to identify what your website goals should be.


  1. Defining website goals

In an era of fancy parallax effects and beautiful graphic design, usability and focus on conversions is often put aside (most of the time, unintentionally). I’ve seen it too many times where a company forgets what is the goal of their website (seriously). A common answer: “just to have a presence on the Web” is a useless reason for having one.

Website goals are what links your business objectives with your website. In other words, they define the purpose of your site. Outlining website goals is one of the step that will help you the most at understanding what you need to track. In the next steps, you will need to set key performance indicators, target metrics and segments; if you don’t identify exactly what you’re trying to achieve with your website, you won’t be able to do any of these steps and you won’t know what to track.


  1. Defining KPIs (key performance indicators)

Now that you have website goals, you need a way to track them. The best way to do this is to find your KPIs. Key performance indicators are metrics that helps you understand how you are doing against your objectives.

When you are going to analyze your analytics reports, KPIs will be your most important metrics.

Without them, how would you measure your website’s goals? How would you know which metrics are important for your business? Make sure they are clearly defined.


  1. Defining your targets

Targets will determine if your KPIs are being successful or not, they are the goals of your KPIs.

So how do you determine if your KPIs are successful or not, or how do you know if 100 signups is good for your company? You set targets! Easy peezy, right?

If you don’t set targets, you will see your KPIs and numbers either increase or decrease, but you won’t know if the change in numbers is a success or failure for your business objectives. Ignoring this step would be comparable to a company that know they want to increase their revenues, but does not set revenue goals. It just doesn’t make sense.


  1. Creating segments

Segments are awesome. They allow you to dive into granular details within your analytics by only seeing the data related to a particular subset of your visitors.

Using segments, you are able to create groups of your focus customer, groups visitors in new markets you want to tackle, or groups of visitors with certain behaviors, and so on. This means that if you want to know precisely how a certain group interacts or converts on your website, you are able to get the data in just a few clicks.

Segments are a bit more advanced, and in many cases, their efficiency relies on your ability to interpret your data. In the planning phase, this is the step that is the most skipped, but if you want to truly dominate at conversion optimization, don’t be afraid of them and make a good use of it.


  1. Finding what to test

This one is obvious, yes, whether you follow the process or not, you will need something to test; however, this is where many will improvise. Over-reliance on other people’s case studies and huge blog posts listing all the elements you could test are part of the problem.

But here’s the thing: you don’t want to test something because you heard it worked for someone else, you want to test what you identified could be improved using data in your analytics systems such as Google Analytics. Using surveys, doing user testing, and similar methods are excellent at uncovering conversion barriers.

Deciding to test anything and everything for the sake of testing won’t lead you to the results you hope, the biggest improvements will come from problems you have identified using data. Hence why I can never emphasize enough on why taking the time to identify the right elements to improve is always worth it.


  1. Prioritizing testing ideas

When you have enough ideas of what you could test, randomly picking one idea to test is not the right approach as it could lead to testing ideas that would require unnecessary amounts of resources. It is important to analyze your ideas and prioritize. I’m a big fan of Chris Goward’s P.I.E. framework for prioritization, but this many other techniques also exists.


  1. Formulating an hypothesis

Conversion rate optimization is science, and guess what? Scientists have been formulating hypothesis for their experiments since the beginning of time.

The same applies when you A/B test, a well-defined hypothesis will allow you to observe in details your findings and identify a potential solution.



The testing phase is the best known and least skipped part of the process. I won’t go into the details of why the first three steps are important; they’re already very obvious:

  1. Selecting the right tools
  2. Creating test variations
  3. Executing your test

But here’s why the other lesser-known steps matters:


  1. Track your on-going test

Tracking your on-going test is extremely important. It could happen that a test you’re running was mistakenly misconfigured, and realizing your data is skewed after two or three weeks of testing is not a surprise you wish to encounter. Not only would this cost you valuable time, but it could also be costly – the faster you find a winning variation, the faster you increase your revenue.




  1. Post-test analysis

After your test is over, you want to ensure all the data is correct and reliable. Make sure you don’t have possible false positive/type 1/type 2 errors or any other possibility of things that could have skewed your data. Don’t forget to use logic, if the numbers appears suspicious, you might want to dig deeper or even do a re-run if necessary.


  1. Implementation

Procrastinators, I’m looking at you. I’ve seen it before, which is why I’m pointing it out: a company runs an A/B test but fails to implement the winning outcome.

This is a must avoid at all cost. If you’ve followed this process, it means you have invested a considerable amount of time into improving your website (which, if done correctly, the ROI should be positive), but if after having done all that work you don’t proceed with implementing your findings, all your efforts are worth nothing. I’d be like running a company that is successful, but abandoning it half-baked for no reason. Doesn’t make sense, doesn’t it?



Conversion rate optimization is not meant to be a one-time thing. On average companies perform between 1 to 5 tests a month, and big retailers such as Amazon can run upward 500 per month!

There will always be something to improve on your website, whether its obvious or not, and not optimizing constantly won’t keep getting you improvements. If you want to constantly increase sales, signups or revenue, keep testing, keep optimizing.


Now that you know the importance of the process and all its steps, before testing any other element you “think” could help your conversion, remember to follow it.

Although this process will require some efforts, following it will save you plenty of misused man-hours, and your optimization work should immediately become more effective.


Are you currently following a similar process when you’re optimizing? If not, will you start after having read this post? Let me know in the comments below.