Billy's Blog: Multivariate Testing & Optimization

A beginner’s mistake is to test every idea with every test. This is the most obvious way of being efficient. If I can test 50 things in a week, why not?

In my experience, efficiency has more to do with careful test design and doing things right the first time, than trying to test everything and rushing the process. By testing a few big ideas quickly and then designing the next test based on those results, you can do a set of small tests and get answers fast without having to risk your page to many bad ideas.

Every test should have specific questions its trying to answer. Not just “What’s the best performing page?” but questions that lead to that. A car salesman doesn’t blindly try every tactic in the book get you to buy a car, a real salesman probes you with a few questions and changes their technique accordingly.

That’s how you should design your tests.

Here’s an example test plan that works for most clients:

• Step 1 (Split Test) – Find an optimal template/design: What template and/or design effectively gets visitors to stick, click and convert? At this stage, you aren’t testing messaging yet, you’re merely re-skinning and moving elements around to find a good design. Some techniques to use are simplifying the page by de-emphasizing unimportant content (shrink company logo, move ads to the bottom of the page) and emphasizing core content (moving 3rd party validation near the call to action) and adding more whitespace to the page to enhance readability. These are in addition to a well done creative design. This test usually has the greatest impact, however it all depends on your current page and the audience. (Read more on template testing)
• Step 2 (Multivariate Test) – Find the biggest converting segment: This test focuses on finding the correct messaging by appealing to different segments that you know and hypothesize visit your page. If your product was Google Apps, you might test appealing to business users and freelancers. Or if you are selling a cell phone, you might test features against benefits.
• Step 3 (Multivariate Test) – Find the perfect way to communicate to the segment: Step 2 points you in the right direction, but this step helps you find the exact place you should be with your page. Use what you learned (freelance messaging won) and try variations on that winning theme to really grab your audience and give them what they want. Also, step 2 may have revealed 2 or more segments that are worth targeting. If you can segment them out, run multiple tests that are customized for each segment, and you’ll raise conversions even higher.

The alternative is to test 50 ideas of which many of the ideas overlap. Why test any ideas that are remotely similar until you know that they work in general? If I go to a dealership wanting a sports car and the dealer offers me 5 colors of minivans, I’m still not going to buy a minivan. Show me 4 types of cars, let me pick the one I like and then we might talk about color.

Let your visitors lead you!

This really is a simple process, but it drives results. Be methodical to be efficient. By course correcting in each test, you get closer and closer to what you need and don’t spend a lot of time testing losing elements. Follow a test plan like this and you’ll get results and learn a lot about your core converting visitors.

I just received a link to an amazing resource from MarketingExperiments, it’s a compilation of great webinar summaries and case studies that they have done. They cover topics from landing page optimization to price testing to PPC and more. While not everything is about testing specifically, all their advice and ideas can be tested, which is why I think you all will find it valuable.

All testing should be carefully designed; it should be focused on best practices and tactics that are predicted to connect with the audience. You should take risks when testing, but they should be calculated risks.

Check it out and soak up some knowledge on optimization and get ideas to test on your site.

Quality PPC and LPO campaigns are key to great conversion rates. If either of them are optimized, you might get good results, but with both of them optimized, your gains are exponential. There are a few pitfalls in optimizing them both though, even with good intentions you may end up confusing your results rather than getting results.

Here are 3 methods to effectively optimize your PPC and landing pages:

1. Do one at a time: Test out your new PPC strategy, but wait until your landing page testing is done. Changing your PPC means you’re changing the audience, both in demographics and expectations. This will impact your landing page testing. Once you find a winning PPC campaign, test the same messaging on your landing page. This is the easiest way to optimize both, but the next two are better ways to go.
2. Do them simultaneously: If you are testing 2 PPC strategies, create 2 separate landing page tests to match the respective campaigns and drive traffic solely to the matching test. This avoids biasing the PPC that better matches your landing page.
3. Segment all the way through: For segments you know you’re going to have, make them go to different landing pages. Test your pages and separately track how each segment performs. Sometimes all your segments respond best to the same landing page, but often times your segments want something different and it’ll show in your testing results. Also, if you’re doing #2 and realize that the ROI is good enough for both campaigns, break it out and optimize them separately.

These are basic, but very effective methods to maximize testing both your PPC and landing pages. If you want to get actual and sustainable results, you have to control as many variables as possible. Only when you can trust your data, will it perform how you expect. Follow any of these methods and you’ll be on your way to higher conversions.

After 4 months, I finally received certification via the *breath in* Marketing Experiments Certification Course on Landing Page Optimization – Subscription Path Track.

If you already follow Marketing Experiments, much of the material they put out for free is discussed in the class (although in greater depth.) Flint McClaughlin, who runs Marketing Experiments, knows testing and optimization very well, but the class could be stronger. Taking the class, training at Widemile and working with clients simultaneously has taught me a lot, very quickly, so as the class went on, I wasn’t learning as much. Those of you who don’t have the benefit of being surrounded by testing pros, probably will get a lot more out of it.

In addition, sometimes the number of conversions for their case studies are quite low, which leads me to question some of the testing numbers. I’m sure they got lifts, but their numbers are a little outrageous at times and, as my boss Frans brings up, seem to not account for seasonality.

Despite all that, I’d definitely recommend it to anyone who wants to get into or needs to learn how to make better landing pages. They offer many other certification classes and while I can’t really say how good the other classes are, I have a strong feeling that they are worthwhile too.

Regardless if you do or don’t want to take the class, you should take the time to learn their conversion index formula. It’s the overarching idea of the class and really helps you think in a systematic way about what should be improved on your pages and funnels. The conversion index formula is:

C is the probability of conversion, so this formula deals with variables that cause visitors to convert or not convert. Here’s the quick rundown of each letter:

• m – motivation of the visitor
• v – your value proposition
• i – incentive to convert
• f – friction of the process
• a – anxiety about converting

I don’t want to go into too much depth, but I will mention that my favorites are incentive and friction. They are together in the equation because they counteract each other. You use incentives to overcome the friction of the page. So offering your visitors a white paper helps them deal with giving up their name and e-mail address to you. Visitors know they are going to get a call or e-mail when they give you info, but you have to give them reasons to give it to you.

An example of this that most of you have probably experienced as an internet user is when you find a great deal on a badly designed website. Even if it’s tough to get through the checkout process (friction), you’re likely to finish it since it is a great deal (incentive).

One of things I’ve learned is that making website changes is rarely a streamlined and easy process. The best situation would obviously be to have good incentives and low friction, but you can’t always improve everything because of office politics, technical reasons, lack of resources or numerous other things. So by using this formula and keeping these things in mind, you have multiple ways to attack problems either offensively (increase incentives, value proposition, focus on user motivation) or defensively (decrease friction and anxiety).

Sometimes people ask me why landing pages are such popular targets for optimization and testing. Why not optimize a home page? A product page?

Actually, we can optimize those kinds of pages, but almost all businesses come to us with a landing page that needs help. Beyond the demand for landing page testing though, is the fact that landing pages inherently are fit for testing. Let me explain.

One of Widemile’s landing pages

The strength of multivariate and split testing is in pulling out the best page possible for what you want to do. With a homepage you have multiple things you want to do (e.g. show your products, company history, customer service, get people to spend X minutes browsing) and with a product page you are typically working with a CMS template for your whole site. Those factors complicate things a bit since you have to figure out what counts as improvement for the homepage and for the product page, you have to work with a CMS system and make changes that improve the majority of products pages using that template.

These things are not impossible to do or even difficult in many situations, but a landing page is usually totally independent of everything and has only one goal.

In a technical sense, a landing page is more simple to deal with. In a measurement sense you only need to improve one metric, the conversion. With a landing page, we don’t have to make copy and creative that works okay for all situations, we simply make copy and creative that is optimal for one situation.

This makes testing really fun and easy, since you can test and find out who your audience really is and what causes them to convert.

Now that I’ve answered this question, in the future I’ll move onto a more interesting post: Why you should optimize everything else too.