« Getting Cultured in Testing | Home | What can your data really tell you? »
5 quick tips to effective A/B and split testing
By Billy | December 5, 2007
While multivariate might be the hottest testing subject, you can’t beat a good split test in certain situations.
If you already know the difference between A/B and split tests, skip this part, otherwise it’s a quick read.
A/B testing is when you test one page then replace it with a new page, so the two versions are running concurrently, one after another. Split testing is when you test two pages at once, where some distribution of traffic is sent to either of the pages simultaneously.
A/B Test
Split Test (note you can split different %’s)
Now for the good stuff! I’ll try to keep this short so let’s start:
- A/B testing is out: Use split testing instead. Split testing is more accurate since it uses the same time period of traffic. Traffic during Halloween is different from traffic during Christmas, so testing one page at one time and one at another will skew your results making a relevant comparison impossible. Use A/B only if you don’t have permission/the capability to do split testing (Google Optimizer is free and allows split testing!)
- Template test = Split test: This is the sweet spot for split tests. Use one if you want to try a new layout/template against your old one or if you want to test two new pages. Here’s my template test primer if you need to brush up.
- One exception, one lesson: During these tests everything should be the same except for one thing on the page. If you try introduce 2 or more changes into a split or a/b test you won’t know which change improved your page. The only time I might have multiple changes is for template tests, where the new template can’t use the previous creative effectively. Still, emulate the creative as closely as possible for the new template.
- Be ready for your next test: Since these tests are easier to execute, you should also have an easier time getting the next test ready to go for when the first one finishes. Make tests ahead of time so that when the current test completes you can flip the switch and quickly get it measured and done with.
- Learn from the first test: You already completed a test, what does that tell you about what you should test for the next time? If a graphic heavy template beat the cleaner template, try testing against an even more graphics heavy template. Find where your customers lie and pinpoint it by seeing what each test tells you. This is a game of Marco Polo. You customers are shouting, “Polo!” with each test, follow them!
Split tests keep it simple and that is its strength. As long as you control anything that might confuse the test (like introducing new content), you can find winners and make a great page. However, after split testing, multivariate testing should be brought in to really pull out more from your page. But that’s a whole other blog….
Related posts
Topics: Methodology, Terminology |
December 16th, 2007 at 6:33 pm
Hi Billy,
Thanks for sharing the wonderful insight, I have a query though, what if you’re doing a test by adding 2 or 3 new call to action buttons to segment the audience or consumers, alongside with the existing call to action button. And lets say if I want to change the message or the design of the new call to action button, how do I measure it in comparison with the existing call to action button ?
Cheers,
deric
December 17th, 2007 at 8:54 am
I’m not quite clear at what you’re trying to do. I think you are saying you are trying to test 2 or 3 call to action buttons that hit on different messaging for various segments. Along with that you want to try different designs for the button. All of this is to be tested against the current button.
In that case, I would recommend multivariate testing because you are trying so many different combinations at once. However, you could do this through many split tests.
If you have enough traffic/know your segments are large enough, I would create separate landing pages for each segment. Then within those landing page segments you could do split tests for them. So segment A would have a split test running, while segment B had its own split test. These would be current button versus new button.
You should test button design separate from messaging. I find messaging to have a much higher impact than button design, so I would test that first, find which one works and then try out button designs in the next split test for that page.
As you can see, that is a lot of work just to test multiple buttons, this is why multivariate testing is much more powerful. You could test this all on one page and find out the best overall page and what segments you have by the effectiveness of different messaging you test. This is why I recommend split testing for templates rather than just one element like a button.
Let me know if this answered your question!
December 18th, 2007 at 9:11 am
Thanks for sharing the wonderful insight Billy, I’m gonna try it out and see how it goes.
cheers,
deric
December 26th, 2007 at 8:08 am
[...] Allows for quicker testing of multiple elements than split or A/B testing [...]