« Why always optimize landing pages? | Home | Every marketer’s New Year’s Resolution »
Multivariate testing: a quick primer
By Billy | December 26, 2007
Want to quickly get up to speed on multivariate testing? This post is designed to help you grasp the basics of multivariate testing so that you can get started talking and even doing your own tests. While its hard to design great tests, it’s easy to get good results with only a little education. I will definitely be teaching more about multivariate testing soon, but get this stuff down first!
Note: Since the industry is new, there isn’t consistency in much of the vocabulary of multivariate testing, so I will try to use generic terms.
What is Multivariate Testing?: Testing multiple versions of a page to determine a set of elements* providing the highest conversion rate.
*Elements can be any content (text or image) on a page, typically hero shots, buttons, button text, headlines and text blocks. Sometimes it can refer to position/layout also.
What happens:
- Visitors come to a page and are shown a random version of the page
- Conversions are tracked based on which page they saw
- Once a statistically significant number of conversions is reached, analysis determines the elements on a page that create the highest conversion rate
Strengths:
- If used correctly, it hones in on correct messaging direction and then exact messaging can be found using further testing
- Allows for quicker testing of multiple elements than split or A/B testing
- Analysis derived from live visitor data
- Proves winning page elements to be better than others
Weaknesses:
- Medium learning curve
- Test results easily ruined by mistakes or poor methodology
- Requires a minimum number of conversions
- Code must be added to web page
- Cookies and JavaScript must be enabled by visitors
Keep in mind:
- Increasing the number of elements being tested increases the time needed for the test.
- Number of conversions over time determines test size and length
- The faster a page gets conversions, the shorter the test will be and/or the more things you can test
- More conversions is better
- A shorter time frame is better than a long time frame, but don’t go shorter than 2 weeks
- Don’t test things that are too similar, look for different segments or messaging to pursue
Other concepts:
- Using it with split testing:
- Use split tests to determine the best layout with a template test. Test layout against layout with the same content (see this article for more advice.)
- Afterwards, use multivariate testing to try out different messaging and refine it with continual tests.
- If you want to try new layouts later on, go back to split testing.
- Advanced: There are two types of multivariate tests: Full factorial and fractional/partial factorial
- Full factorial means every version is shown. Meaning if you have 4 headlines and 4 buttons, that is 4×4=16 combinations, so 16 pages are used in the test.
- Partial factorial is when only a portion of the total possible combinations are shown to visitors. This relies on statistical formulas and algorithms to determine the influence of the various factors since not every page combination is shown.
- Full factorial takes a much longer time, so partial factorial allows for more testing in a quicker time frame.
There’s still a lot more to teach, even about some of the things mentioned here, but I hope this is enough to get you off the ground and really digging into testing. Google has their free tool if you want to try out multivariate testing.
Related posts
Topics: Terminology |
March 28th, 2008 at 11:05 am
Hi Billy,
Thank you for providing great information on multivariate testing. I am searching for information on template tests as well as on fractional factorial multivariate tests. Can you reference and resource that provide extend info on these two topics, in terms of formulas, algorithms and etc. Also, you mention about the Landing Page Optimization Certification program with MarketingExperience, does this program cover multivariate testing (including template testing)?
Thanks in advance for your response.
Connie
April 1st, 2008 at 9:55 am
Hi Connie,
I talk about template testing here:
http://testingblog.widemile.com/2007/12/05/5-quick-tips-to-effective-ab-and-split-testing/
If you’re looking for formulas and algorithms that’s much more complicated and falls within statistics. Unless you’re looking to build a tool from scratch, you should look towards existing testing tools such as Google Web Optimizer or what Widemile offers.
I discuss fractional factorial test design here:
http://testingblog.widemile.com/2008/01/28/why-google-optimizer-is-free-its-old-and-slow/
The LPO Certification does not cover template testing or multivariate testing, it just covers LPO best practices and gives you solid ideas to test. I believe they also have started a class specifically about testing, but have not taken it myself.
I hope this helps,
Billy