Billy's Blog: Multivariate Testing & Optimization

I just received a link to an amazing resource from MarketingExperiments, it’s a compilation of great webinar summaries and case studies that they have done. They cover topics from landing page optimization to price testing to PPC and more. While not everything is about testing specifically, all their advice and ideas can be tested, which is why I think you all will find it valuable.

All testing should be carefully designed; it should be focused on best practices and tactics that are predicted to connect with the audience. You should take risks when testing, but they should be calculated risks.

Check it out and soak up some knowledge on optimization and get ideas to test on your site.

Note: This is the second post of a 3 part series, each focusing on one type of test result that is tough to deal with. Read the first article on highly mixed data.

As an optimization analyst, this is probably the hardest result to bring to a client. Oddly enough, it actually is favorable to part 1’s highly mixed data and part 3. I am talking about optimization that determines that the original page is better than the tested variations.

How does this happen?
Sometimes a page just gets it right. How would you change Google? I looked for a few variations and came across one by Andy Rutledge and another by Valacar. They both are beautiful designs and a lot of thought were put into them, but at the same time, would they really make Google more profitable? It’s definitely a tough sell and there is a big challenge in improving this type of page.

The goal is for users to search. Yes, they want users to click on ads eventually, but there’s not a whole lot they can do for ad clicks on the homepage. The best they can do is get users to search as fast as possible. So would a redesign make it more usable and readable? Maybe. To a level that it would increase their revenues? That’s tough to say.

The more simple the goals of the page, the less information and messaging the users needs, the more likely that the page will be difficult to optimize.

What can you do to prevent this?
Be careful when choosing a page to test. Find a page where the user will take some time to look at what is going on. This is another reason why most landing pages are great places to optimize, because users naturally need to be introduced to the product and sold on why to convert.

The logical thing to do would be to simply refrain from testing pages that seem to be performing well, but this is rarely a good rule. Unless it is performing well because of a lot of testing, then you don’t really know if a page is performing well or not (see my post on conversion rates.) Testing always brings surprises and personal judgment is no replacement for a test; a good looking page can perform poorly and a page with subpar creative can perform great.

What can you do if this happens?

Because of the above reasons, you may actually plan for this scenario to occur. Many people believe redesigning an old page will provide improvement, but what if it is old and performing well? In that case, you may plan to try to improve but not expect to beat the old version.

In any case, if your original page wins, then you have confirmation of your page’s success. It is unlikely that all possible improvements were tested in one test run though, so it may take a few more runs to really confirm its solidarity, but the page has won against the initial best ideas and that is an achievement.

This lesson tells you that you can move on and that is progress in itself.

Moving forward, I would try drastically different approaches, either in layout or design and testing around offers. Otherwise, I would apply the successful original page to tests for other areas of your site.

I have to be honest when I say that this rarely ever happens. Almost every page has room for improvement at every step of the conversion funnel.

Whew, I will try to get the third and toughest optimization result next week.

CC photo credit: philosophygeek

Edit: I fixed all the links in this post.  Copy and pasting is getting the best of me!

I recently came across a few great posts that I enjoyed and wanted to pass onto you all. The first is from Tim Ash, who has written a great book on Landing Page Optimization. One of his more recent entries discusses how to write effective copy to increase conversions.

One of my favorite bloggers, Avinash Kaushik tells marketers to embarrass their managers in order to succeed at their campaigns. Testing tops that list of course, but his other techniques are great methods at “working the system.”

Lastly, Lenny de Rooy, wrote a guest post at SEO Scoop about 5 misconceptions of Google Web Optimizer. It goes slightly beyond just GWO itself and into testing methodology

A big mistake in testing is to overlook variables inside and outside of the test that impact results. In an ideal test, the only variables would be the ones you are testing on your page. That usually isn’t possible though, but as long as you account for them in your analysis, you will get correct and actionable information.

If you test a seasonal page, then the optimal page you get for that season, probably won’t perform when the season ends. By not paying attention to those kind of variables, you are setting yourself up into thinking you’ve found the optimal page. The same type of mistake is made by grouping e-mail, print, SEM campaigns and event traffic, unless you know they react the same to your changes.

Even within segments, there might be more segments to uncover. Your only limitation should be traffic; don’t segment so granular that you can’t run a decent sized test in a decent amount of time.

One of my clients doesn’t get a lot of traffic, but the traffic he does get is very distinct. One converts in the single digits and the other converts in the teens. Although combining them would get me more data, it would be very confused data since they convert so differently.

A few things to look out for:

• The ad or offers visitors see beforehand
• Interactions between your factors (if you aren’t testing interactions)
• Technical problems
• Problems that occur before or after the tested page

A note about the last bullet, the problems can range from a technical problem to a problem with the overall funnel. If people get different experiences in the funnel that drastically impact whether they convert or not, it can add a noise to your test. Some examples are different checkout processes for registered and non-registered users or users being inelligible for service.

The purpose of testing is to find out if a certain element performs well under the conditions you provide. If you aren’t paying attention to all the conditions, then the results you derive will be incorrect without you knowing.

Online testing is a bit different from other marketing data. It uses live traffic to find out what works. Analytics is the same, measuring what’s occurring at the moment. So why is that important? Well you can infer all you want from surveys, usability studies and demographics, but in the end you can’t argue against what real users are doing.

Avinash Kaushik, a popular analytics blogger, summed up the juiciest bits of a presentation by Jim Novo at eMetrics. In it, Jim asked, “What data yields insights that can be actioned the most?” The answer:

“[A]ctionability, relevance of insights that can be actioned decreased as you go down the slide”

He makes the point that the farther away you get from the top of the pyramid, the harder it is to accurately predict your users actions. Yet often times too much value is put into the bottom levels of the pyramid. Even when marketers do test and measure actual behavior, they go about it the wrong way because they stick to all this other data too much and end up testing things that are all alike, defeating the purpose of testing.

Think about it, can you really tell if a red button will work better than a blue button if all you have are demographics? There are places for all of these types of data, but there should not be a fear of actual behavioral data. Yes we are using live traffic, yes the data is driven by technology (online visitors, javascript, cookies, rather than people filling out a survey), but those numbers tell a story unlike any other data.

Make the most out of all types of your data, but don’t die by one or the other. Use what’s best for every situation, but realize that you will never know you are right until you test it out.