Billy's Blog: Multivariate Testing & Optimization

I just received a link to an amazing resource from MarketingExperiments, it’s a compilation of great webinar summaries and case studies that they have done. They cover topics from landing page optimization to price testing to PPC and more. While not everything is about testing specifically, all their advice and ideas can be tested, which is why I think you all will find it valuable.

All testing should be carefully designed; it should be focused on best practices and tactics that are predicted to connect with the audience. You should take risks when testing, but they should be calculated risks.

Check it out and soak up some knowledge on optimization and get ideas to test on your site.

Note: This is the second post of a 3 part series, each focusing on one type of test result that is tough to deal with. Read the first article on highly mixed data.

As an optimization analyst, this is probably the hardest result to bring to a client. Oddly enough, it actually is favorable to part 1’s highly mixed data and part 3. I am talking about optimization that determines that the original page is better than the tested variations.

How does this happen?
Sometimes a page just gets it right. How would you change Google? I looked for a few variations and came across one by Andy Rutledge and another by Valacar. They both are beautiful designs and a lot of thought were put into them, but at the same time, would they really make Google more profitable? It’s definitely a tough sell and there is a big challenge in improving this type of page.

The goal is for users to search. Yes, they want users to click on ads eventually, but there’s not a whole lot they can do for ad clicks on the homepage. The best they can do is get users to search as fast as possible. So would a redesign make it more usable and readable? Maybe. To a level that it would increase their revenues? That’s tough to say.

The more simple the goals of the page, the less information and messaging the users needs, the more likely that the page will be difficult to optimize.

What can you do to prevent this?
Be careful when choosing a page to test. Find a page where the user will take some time to look at what is going on. This is another reason why most landing pages are great places to optimize, because users naturally need to be introduced to the product and sold on why to convert.

The logical thing to do would be to simply refrain from testing pages that seem to be performing well, but this is rarely a good rule. Unless it is performing well because of a lot of testing, then you don’t really know if a page is performing well or not (see my post on conversion rates.) Testing always brings surprises and personal judgment is no replacement for a test; a good looking page can perform poorly and a page with subpar creative can perform great.

What can you do if this happens?

Because of the above reasons, you may actually plan for this scenario to occur. Many people believe redesigning an old page will provide improvement, but what if it is old and performing well? In that case, you may plan to try to improve but not expect to beat the old version.

In any case, if your original page wins, then you have confirmation of your page’s success. It is unlikely that all possible improvements were tested in one test run though, so it may take a few more runs to really confirm its solidarity, but the page has won against the initial best ideas and that is an achievement.

This lesson tells you that you can move on and that is progress in itself.

Moving forward, I would try drastically different approaches, either in layout or design and testing around offers. Otherwise, I would apply the successful original page to tests for other areas of your site.

I have to be honest when I say that this rarely ever happens. Almost every page has room for improvement at every step of the conversion funnel.

Whew, I will try to get the third and toughest optimization result next week.

CC photo credit: philosophygeek

Note: This is the first post of a 3 part series, each focusing on one type of test result that is tough to deal with.

There are 3 types of optimization results that people never look forward to getting.

Unfortunately, anyone who runs enough tests will run into these situations. In the following 3 posts I will go over the 3 situations and outline how they happen and how to prevent and handle them.

This first post is about tests with highly mixed data.

How does this happen?
Typically mixed test data occurs when events out of your control, or you forget to control, impact your test and skew your test results.

Your average traffic finds your tested page either through search, browsing around your website or through your planned advertising campaigns. However they get to the page, tests are designed (or should be designed!) based on how visitors will get there.

The problem arises when, outside of the scope of those involved in testing or because of some oversight, a new type of traffic is driven to the page without any preparation being done for that traffic. While traffic is good for your sample size, it is bad because those new visitors are coming in with totally different motivations, assumptions and knowledge. This means they probably will react differently to your tested elements than the traffic you were driving to it originally.

Most often this happens to me because of a new marketing push, such as an e-mail blast, new display ads or promotions at a trade show. This can happen even more unexpectedly if some outside party drives a lot of traffic to your page. A news story or blog review that innocently links to your page, can suddenly becomes a big source of new traffic.

What can you do to prevent this?
The first step is to spread the knowledge around your company that this testing is going on and that anything that may impact the page and its visitors should be run by the optimization team first.

Next, always segment out significant traffic and track it separately. If you segment, the worst case scenario is that they perform the same as your current traffic and you did a little extra work. The alternative is having to trudge through your data, trying to separate the 2 types of traffic and possibly having to restart the test if you can’t separate them out.

Lastly, be aware. Watch your data and look for big changes. If you see something strange or a sudden shift, try to find a cause. It usually will be nothing, but if you do find something, quickly build a segment for it. Even if the “new” traffic has already started hitting your page, a segment should be setup as soon as possible.

What can you do if this happens?

I would still try to segment the data in any way possible. Even taking certain days/time out of your data, may be enough to salvage your results. Do your analysis with and without those days and see if the optimal page changes. You should take extreme care when doing this still though and make sure you have statistically relevant results.

My next solution is just to restart the test. Testing is about continual growth and you can’t always get what you want out of every test. Be happy that you got some extra traffic and try to take precautions to take it into account, or prevent it, the next time around.

Let me know if you’ve ever run into these problems before and how you handled them. Look out for part 2 of this series in the next week.

Photo Source (under CC)