Billy's Blog: Multivariate Testing & Optimization

You and your competitor’s all have the same landing pages.  You have a hero shot of the product, a big call to action button and short, punchy copy.  Or maybe you’re already ahead of your competitors and have run a few tests on your page, picking up more conversions on the way.  In either situation, you’ll eventually hit a wall and struggle to get additional lift.  So how do you continue to improve?

Go for broke.  Try something you’ve never tried before.  It might end up being a total failure, but it also might give you the lift you want.

The gamble you make with optimization can end in 2 ways:

• You lose X amount of conversions over the week or two that the test is running
• You gain X amount of conversions for the effective lifetime of the page

The possible upside dwarfs the downside by a large margin and, either way, you learn something new and can optimize the next test more successfully based on what you learned.

Luckily, with skill and experience, the risks of testing are minimized, however beating a strong page is never easy or guaranteed.  But when you do find something new that works or see that your current page still is a champ, you can rest assured that you’re doing all you can to drive conversions.

Note: This is the second post of a 3 part series, each focusing on one type of test result that is tough to deal with. Read the first article on highly mixed data.

As an optimization analyst, this is probably the hardest result to bring to a client. Oddly enough, it actually is favorable to part 1’s highly mixed data and part 3. I am talking about optimization that determines that the original page is better than the tested variations.

How does this happen?
Sometimes a page just gets it right. How would you change Google? I looked for a few variations and came across one by Andy Rutledge and another by Valacar. They both are beautiful designs and a lot of thought were put into them, but at the same time, would they really make Google more profitable? It’s definitely a tough sell and there is a big challenge in improving this type of page.

The goal is for users to search. Yes, they want users to click on ads eventually, but there’s not a whole lot they can do for ad clicks on the homepage. The best they can do is get users to search as fast as possible. So would a redesign make it more usable and readable? Maybe. To a level that it would increase their revenues? That’s tough to say.

The more simple the goals of the page, the less information and messaging the users needs, the more likely that the page will be difficult to optimize.

What can you do to prevent this?
Be careful when choosing a page to test. Find a page where the user will take some time to look at what is going on. This is another reason why most landing pages are great places to optimize, because users naturally need to be introduced to the product and sold on why to convert.

The logical thing to do would be to simply refrain from testing pages that seem to be performing well, but this is rarely a good rule. Unless it is performing well because of a lot of testing, then you don’t really know if a page is performing well or not (see my post on conversion rates.) Testing always brings surprises and personal judgment is no replacement for a test; a good looking page can perform poorly and a page with subpar creative can perform great.

What can you do if this happens?

Because of the above reasons, you may actually plan for this scenario to occur. Many people believe redesigning an old page will provide improvement, but what if it is old and performing well? In that case, you may plan to try to improve but not expect to beat the old version.

In any case, if your original page wins, then you have confirmation of your page’s success. It is unlikely that all possible improvements were tested in one test run though, so it may take a few more runs to really confirm its solidarity, but the page has won against the initial best ideas and that is an achievement.

This lesson tells you that you can move on and that is progress in itself.

Moving forward, I would try drastically different approaches, either in layout or design and testing around offers. Otherwise, I would apply the successful original page to tests for other areas of your site.

I have to be honest when I say that this rarely ever happens. Almost every page has room for improvement at every step of the conversion funnel.

Whew, I will try to get the third and toughest optimization result next week.

CC photo credit: philosophygeek

Every test teaches something. Almost every campaign test has the goal of raising conversion rates, but really the power of testing is in answering questions. It is just a happy coincidence that it raises your conversions at the same time.

Here are a few of the questions that can be answered with testing:

1. Is giving away a free widget worth while? Would a cheaper widget perform better?
2. Does promoting a few key products work better than promoting them all equally?
3. How much should this product be priced?
4. Does e-mail traffic respond to the same things as SEO and advertising traffic?
5. What benefits are consumers interested in?
6. Will a discount offer make up in conversions, what is lost in revenue per sale?
7. Do banner ads reduce conversions?
8. Is an extra learn more page more effective than a longer page? Is extra information even necessary?

Answering these questions is pretty easy, but you have to think about them beforehand. Design your test so that it asks questions and offers answers. Figure out what questions you want to ask and based on those, give a few separate and distinct answers in the form of variations of your page.

The easy ones are the ROI questions, e.g. giving away widgets, discounts, banner ads, and pricing. Just include them on or off in the test and/or with different amounts. At the end, do a ROI analysis comparing the conversion rates of each variation you test.

If you are wondering if one group of traffic responds differently to a campaign than another, segmentation is what you need. Separately track those segments, but don’t forget to design your test with answers too. Make your variations appeal to different audiences and your segments will be pulled towards the one they like the most with their conversions as proof.

If you want to learn what your customers are looking for, do some market research through testing. Test different types of benefits and see which get the visitors to convert, e.g. technical (this camera contains 1 gb of memory) versus lifestyle (this camera stores 100’s of wedding and birthday pictures).

You can test these ideas in an A/B split test or a multivariate test, but a multivariate test is much quicker and will allow you to test multiple questions and answers simultaneously.

Testing gives you a lot of answers and the better you design those questions into your tests, the more sense those answers will make of your data. Conversion rates are always important, but focusing solely on them won’t get you great results time after time. Think about testing in terms of learning more about your audience and you will find continual improvement in your campaigns.

Would you believe that? And if it were true, would it really mean anything to you? It shouldn’t.

I get asked this question fairly often and at first glance it seems like a logical question to ask, but really the focus should be elsewhere.

From my experience, conversion rates range from less than a percent all the way up to 30% or more. Does knowing that help me optimize my clients’ pages? No. Every page has so many variables internally and externally that it is very difficult and nonsensical to worry about the average conversion rate.

The goal of your page, differences between your product/service against your competition, target you’re trying to reach, avenues you advertise and numerous other factors all effect your conversion rates. A competitor having a higher conversion rate than you, does not mean you’re doing something wrong. Set the baseline for yourself and keep improving it. That’s how marketers should approach their conversion rate.

If you’re testing, you’ll find out if you’re campaign is performing suboptimal and find out what the optimal is at the same time.

Pretty amazing huh?

I don’t tell clients I’m going to get their conversion rates above industry averages, I tell them that I’m going to make their campaign as successful as possible. Do that and you’ll be ahead of competition and ahead of where you were when you first started.