Billy's Blog: Multivariate Testing & Optimization

A beginner’s mistake is to test every idea with every test. This is the most obvious way of being efficient. If I can test 50 things in a week, why not?

In my experience, efficiency has more to do with careful test design and doing things right the first time, than trying to test everything and rushing the process. By testing a few big ideas quickly and then designing the next test based on those results, you can do a set of small tests and get answers fast without having to risk your page to many bad ideas.

Every test should have specific questions its trying to answer. Not just “What’s the best performing page?” but questions that lead to that. A car salesman doesn’t blindly try every tactic in the book get you to buy a car, a real salesman probes you with a few questions and changes their technique accordingly.

That’s how you should design your tests.

Here’s an example test plan that works for most clients:

• Step 1 (Split Test) – Find an optimal template/design: What template and/or design effectively gets visitors to stick, click and convert? At this stage, you aren’t testing messaging yet, you’re merely re-skinning and moving elements around to find a good design. Some techniques to use are simplifying the page by de-emphasizing unimportant content (shrink company logo, move ads to the bottom of the page) and emphasizing core content (moving 3rd party validation near the call to action) and adding more whitespace to the page to enhance readability. These are in addition to a well done creative design. This test usually has the greatest impact, however it all depends on your current page and the audience. (Read more on template testing)
• Step 2 (Multivariate Test) – Find the biggest converting segment: This test focuses on finding the correct messaging by appealing to different segments that you know and hypothesize visit your page. If your product was Google Apps, you might test appealing to business users and freelancers. Or if you are selling a cell phone, you might test features against benefits.
• Step 3 (Multivariate Test) – Find the perfect way to communicate to the segment: Step 2 points you in the right direction, but this step helps you find the exact place you should be with your page. Use what you learned (freelance messaging won) and try variations on that winning theme to really grab your audience and give them what they want. Also, step 2 may have revealed 2 or more segments that are worth targeting. If you can segment them out, run multiple tests that are customized for each segment, and you’ll raise conversions even higher.

The alternative is to test 50 ideas of which many of the ideas overlap. Why test any ideas that are remotely similar until you know that they work in general? If I go to a dealership wanting a sports car and the dealer offers me 5 colors of minivans, I’m still not going to buy a minivan. Show me 4 types of cars, let me pick the one I like and then we might talk about color.

Let your visitors lead you!

This really is a simple process, but it drives results. Be methodical to be efficient. By course correcting in each test, you get closer and closer to what you need and don’t spend a lot of time testing losing elements. Follow a test plan like this and you’ll get results and learn a lot about your core converting visitors.

Quality PPC and LPO campaigns are key to great conversion rates. If either of them are optimized, you might get good results, but with both of them optimized, your gains are exponential. There are a few pitfalls in optimizing them both though, even with good intentions you may end up confusing your results rather than getting results.

Here are 3 methods to effectively optimize your PPC and landing pages:

1. Do one at a time: Test out your new PPC strategy, but wait until your landing page testing is done. Changing your PPC means you’re changing the audience, both in demographics and expectations. This will impact your landing page testing. Once you find a winning PPC campaign, test the same messaging on your landing page. This is the easiest way to optimize both, but the next two are better ways to go.
2. Do them simultaneously: If you are testing 2 PPC strategies, create 2 separate landing page tests to match the respective campaigns and drive traffic solely to the matching test. This avoids biasing the PPC that better matches your landing page.
3. Segment all the way through: For segments you know you’re going to have, make them go to different landing pages. Test your pages and separately track how each segment performs. Sometimes all your segments respond best to the same landing page, but often times your segments want something different and it’ll show in your testing results. Also, if you’re doing #2 and realize that the ROI is good enough for both campaigns, break it out and optimize them separately.

These are basic, but very effective methods to maximize testing both your PPC and landing pages. If you want to get actual and sustainable results, you have to control as many variables as possible. Only when you can trust your data, will it perform how you expect. Follow any of these methods and you’ll be on your way to higher conversions.

Want to quickly get up to speed on multivariate testing? This post is designed to help you grasp the basics of multivariate testing so that you can get started talking and even doing your own tests. While its hard to design great tests, it’s easy to get good results with only a little education. I will definitely be teaching more about multivariate testing soon, but get this stuff down first!

Note: Since the industry is new, there isn’t consistency in much of the vocabulary of multivariate testing, so I will try to use generic terms.

What is Multivariate Testing?: Testing multiple versions of a page to determine a set of elements* providing the highest conversion rate.

*Elements can be any content (text or image) on a page, typically hero shots, buttons, button text, headlines and text blocks. Sometimes it can refer to position/layout also.

What happens:

• Visitors come to a page and are shown a random version of the page
• Conversions are tracked based on which page they saw
• Once a statistically significant number of conversions is reached, analysis determines the elements on a page that create the highest conversion rate

Strengths:

• If used correctly, it hones in on correct messaging direction and then exact messaging can be found using further testing
• Allows for quicker testing of multiple elements than split or A/B testing
• Analysis derived from live visitor data
  • Proves winning page elements to be better than others

Weaknesses:

• Medium learning curve
  • Test results easily ruined by mistakes or poor methodology
• Requires a minimum number of conversions
• Code must be added to web page
• Cookies and JavaScript must be enabled by visitors

Keep in mind:

• Increasing the number of elements being tested increases the time needed for the test.
• Number of conversions over time determines test size and length
• The faster a page gets conversions, the shorter the test will be and/or the more things you can test
• More conversions is better
• A shorter time frame is better than a long time frame, but don’t go shorter than 2 weeks
• Don’t test things that are too similar, look for different segments or messaging to pursue

Other concepts:

• Using it with split testing:
  • Use split tests to determine the best layout with a template test. Test layout against layout with the same content (see this article for more advice.)
  • Afterwards, use multivariate testing to try out different messaging and refine it with continual tests.
  • If you want to try new layouts later on, go back to split testing.
• Advanced: There are two types of multivariate tests: Full factorial and fractional/partial factorial
  • Full factorial means every version is shown. Meaning if you have 4 headlines and 4 buttons, that is 4×4=16 combinations, so 16 pages are used in the test.
  • Partial factorial is when only a portion of the total possible combinations are shown to visitors. This relies on statistical formulas and algorithms to determine the influence of the various factors since not every page combination is shown.
  • Full factorial takes a much longer time, so partial factorial allows for more testing in a quicker time frame.

There’s still a lot more to teach, even about some of the things mentioned here, but I hope this is enough to get you off the ground and really digging into testing. Google has their free tool if you want to try out multivariate testing.

Sometimes people ask me why landing pages are such popular targets for optimization and testing. Why not optimize a home page? A product page?

Actually, we can optimize those kinds of pages, but almost all businesses come to us with a landing page that needs help. Beyond the demand for landing page testing though, is the fact that landing pages inherently are fit for testing. Let me explain.

One of Widemile’s landing pages

The strength of multivariate and split testing is in pulling out the best page possible for what you want to do. With a homepage you have multiple things you want to do (e.g. show your products, company history, customer service, get people to spend X minutes browsing) and with a product page you are typically working with a CMS template for your whole site. Those factors complicate things a bit since you have to figure out what counts as improvement for the homepage and for the product page, you have to work with a CMS system and make changes that improve the majority of products pages using that template.

These things are not impossible to do or even difficult in many situations, but a landing page is usually totally independent of everything and has only one goal.

In a technical sense, a landing page is more simple to deal with. In a measurement sense you only need to improve one metric, the conversion. With a landing page, we don’t have to make copy and creative that works okay for all situations, we simply make copy and creative that is optimal for one situation.

This makes testing really fun and easy, since you can test and find out who your audience really is and what causes them to convert.

Now that I’ve answered this question, in the future I’ll move onto a more interesting post: Why you should optimize everything else too.

Ready for the holiday season? My boss, Frans Keylard (Director of Optimization), wrote up a 3 pager on “5 short tips to maximize your holiday campaign” (download PDF). I summarized it in my own words below, but check the PDF for more detail on how to get your conversions up during the holidays or any season!

Monthly sales increase as much as 20% for many online retailers in December and so having a good site now is more important than any other time. Optimization can drive up your sales beyond the seasonal increase in the short term and, in the long term, helps to add even more lifetime customers to your business. With that in mind, here’s 5 short tips to ramp up your site.

1. Seasons Matter: Change your site to match the season, we call it “seasonal-tuning.” Don’t let your site look stale; you don’t want Halloween colors and images for Christmas time. Matching what your customers are actively seeking will really drive your campaigns to success. This may not always work though, so make sure to test what has worked previously against new seasonal images. Lastly, even seasonal images need testing, so try different messaging in each, e.g. product shots, giving the product to another, and people using the product.
2. Know your existing customers: Reach out to existing customers, they are the most qualified audience and you should know them better than anyone else. Create offers that have a lot of perceived value to them. Also, use offers for people who don’t follow through the whole purchase process and try to pull them back in. They are interested but just need a reason to bite. Remember to know both what customers buy and why too. If you’re best selling chairs are shown in an office setting, when your customers are really home business users, then you might lose conversions. Use a home office image instead and get more relevancy to your items.
3. Find a new audience: Seasonality doesn’t just go for your web pages, apply them to your PPC text and banners ads and you’ll get new visitors quickly. Think holidays, turkeys, gift-giving, family, anything that resonates with your audience and your products. Include some seasonal offers for good measure too. You can do SEO around these words too, but that takes future planning. A good way to setup seasonal SEO for next year is by looking at what PPC and SEO worked this year, so make sure you don’t lose that valuable information.
4. Offers over branding: Don’t get too caught up in branding, just sell the product and offer. People are under pressure to buy nowand if your focus isn’t on helping them to do that, they’ll find someone else. You can build your brand later and have a reason to if they bought from you!
5. Build seasonal landing pages and optimize them with multivariate testing: Building a seasonal landing page is easy to do and optimizing that page is a must during seasonal periods. We optimized an online recipe search over the summer and found the best summer imagery, but once winter came around we changed the type of food and got a 30% sustained lift over the best performing summer image.

Stores change their decorations every season, so make sure to do that with your online store front. Give people a reason to stay at your site, even if you think it is small, it could have a huge impact. Finally, never forget to test and tune your changes every season, every year. Good luck!