Running an AB test correctly is extremely important. Each change you make can alter the results drastically and have an impact on your data analysis and completely change your next steps.
A live example:
Last year I decided to run an “AB test” and see what cats prefer: sitting in a plate or a box.
As you can see (in this really bad quality image on the left), I left the cat 2 options on the table; either a box or a plate and put a camera to see where he sat and for how long. The results showed he sat in both, but sat for much longer in the box.
Was this a conclusive test? Obviously not (but it was fun at the time). There are many reasons why this test wasn’t conclusive at all.
There are many things you should make sure of, check or prepare before starting a test, here’s a list of 5 AB test mistakes you want to avoid before your next AB test:
Mistake #1: You’re not sure where to start:
Deciding where to start you testing is one of the most important parts of your optimization efforts. It is important to look at the funnel as a whole and understand that even changes on a simple banner can have an impact on your ROI. When planning your conversion test, start with a strategy. Changing minor things like the call to action or copy can make a (small) difference but it will be very hard to learn anything from it. Start by testing strategies and make your way down the funnel. By testing bigger concepts and larger changes that are a part of a whole strategy you will be able to learn much more from each test and then scale down to the impact of the button or color. Check out an article we published about the 3 important elements of your conversion optimization strategy.
Mistake #2: You’re not giving your test enough time:
This is actually the key conversation we have with every new client. Optimization requires time, and more importantly tests need to be conclusive and have statistical significance. There are many things that can impact the results, anything from the time of day, a holiday or technical issue with the site. Results go up and go down very quickly, and in order to move on to the next stage you need to be sure that the results are conclusive. Two reasons you want to give your test time:
-
You can learn from any test and especially from a failed test.
-
Results change rapidly and need time to even out. Check out this case study which was showing failed results in the first 3 days and ended with an 86% increase in revenue.
Mistake #3: You’re not calculating the sample size before the test
Following the previous mistake, we need our tests to be statistically significant. To do this, you need to calculate the amount of traffic needed for your test to be significant. Calculating the sample size you need will give you the confidence to know whether the test you’re running is one you can actually learn from and optimize. Visual web optimizer has a nice tool that can help you calculate your needed sample size here.
Mistake #4: You’re running multiple tests with overlapping traffic
One thing that’s important to remember is that the tests we run are a way for us to learn more about our funnel and customers. The tests we run need to be easy to understand and analyze. A common mistake people make is running multiple tests at the same time with the same traffic. Doing this makes it harder to get conclusions from the tests and understand what worked or what didn’t. Running multiple tests is a great option (and recommended) if you have the traffic for it, but if you want to learn from your tests be sure to run it with different traffic.
Mistake #5: You’re not tracking the right goals
When running conversion optimization tests it’s important to track the right goals and more importantly, track them all. When you’re only tracking one goal (purchase for example) you might be getting statistics for the end results – whether someone purchased or not, but you’re losing out on the entire funnel stats and you’re missing the leaks. Place goals across the entire funnel to see where the leaks are. Different changes throughout the funnel can increase one part but lower another, for example changing the call to action button can increase downloads in a funnel but decrease installations. All this information should be visible to you so you can better analyze your funnel and continue optimizing without damaging a different part of it.
What’s Next?
Ab testing is the first step in a long pathway of optimization. Doing it right means you’re continuously analyzing the data and optimizing your funnel. Our favorite quote in the office is “Never stop testing, and your advertising will never stop improving.” – by David Ogilvy
Analyzing the data is one thing, knowing what to do with it is another. Don’t be hesitant to talk to conversion optimization companies who can help you analyze your data and optimize. There are many great tools out there who do AB testing that really give you a clear view of your results, but similar to needing a platform to run your tests and help analyze the results you may need someone to help you take the next step and know what to test next.
We’d love to hear from you: What other common mistakes should we avoid in AB testing?
5 Fatal AB Test Mistakes You Must Avoid 5.00/5 (100.00%) 4 votes
Related Posts
5 Fatal AB Test Mistakes You Must Avoid
Running an AB test correctly is extremely important. Each change you make can alter the results drastically and have an impact on your data analysis and completely change your next steps.
A live example:
As you can see (in this really bad quality image on the left), I left the cat 2 options on the table; either a box or a plate and put a camera to see where he sat and for how long. The results showed he sat in both, but sat for much longer in the box.
Was this a conclusive test? Obviously not (but it was fun at the time). There are many reasons why this test wasn’t conclusive at all.
There are many things you should make sure of, check or prepare before starting a test, here’s a list of 5 AB test mistakes you want to avoid before your next AB test:
Mistake #1: You’re not sure where to start:
Mistake #2: You’re not giving your test enough time:
This is actually the key conversation we have with every new client. Optimization requires time, and more importantly tests need to be conclusive and have statistical significance. There are many things that can impact the results, anything from the time of day, a holiday or technical issue with the site. Results go up and go down very quickly, and in order to move on to the next stage you need to be sure that the results are conclusive. Two reasons you want to give your test time:
You can learn from any test and especially from a failed test.
Results change rapidly and need time to even out. Check out this case study which was showing failed results in the first 3 days and ended with an 86% increase in revenue.
Mistake #3: You’re not calculating the sample size before the test
Following the previous mistake, we need our tests to be statistically significant. To do this, you need to calculate the amount of traffic needed for your test to be significant. Calculating the sample size you need will give you the confidence to know whether the test you’re running is one you can actually learn from and optimize. Visual web optimizer has a nice tool that can help you calculate your needed sample size here.
Mistake #4: You’re running multiple tests with overlapping traffic
One thing that’s important to remember is that the tests we run are a way for us to learn more about our funnel and customers. The tests we run need to be easy to understand and analyze. A common mistake people make is running multiple tests at the same time with the same traffic. Doing this makes it harder to get conclusions from the tests and understand what worked or what didn’t. Running multiple tests is a great option (and recommended) if you have the traffic for it, but if you want to learn from your tests be sure to run it with different traffic.
Mistake #5: You’re not tracking the right goals
When running conversion optimization tests it’s important to track the right goals and more importantly, track them all. When you’re only tracking one goal (purchase for example) you might be getting statistics for the end results – whether someone purchased or not, but you’re losing out on the entire funnel stats and you’re missing the leaks. Place goals across the entire funnel to see where the leaks are. Different changes throughout the funnel can increase one part but lower another, for example changing the call to action button can increase downloads in a funnel but decrease installations. All this information should be visible to you so you can better analyze your funnel and continue optimizing without damaging a different part of it.
What’s Next?
Analyzing the data is one thing, knowing what to do with it is another. Don’t be hesitant to talk to conversion optimization companies who can help you analyze your data and optimize. There are many great tools out there who do AB testing that really give you a clear view of your results, but similar to needing a platform to run your tests and help analyze the results you may need someone to help you take the next step and know what to test next.
We’d love to hear from you: What other common mistakes should we avoid in AB testing?
Related Posts