Optimization Glossary
By Billy | May 15, 2008
Someone at the office has put together a great glossary, so I modified it slightly and have posted the glossary as its own page (it has a tab dedicated to it above now.) It is in a usable but not optimal form right now, so I’ll be updating it every now and then. I have also decided to add expanded definitions, in the form of separate pages dedicated to a single word. The first word to be done was “experiment.” Please check it out and let me know what you think.
In the past, I have stepped away from using technical language and jargon but, with this glossary, I will begin using the language I use at the office. My hope is to acclimate others and help them understand the terminology used by myself and others at Widemile and around the industry.
Topics: Site News, Terminology | No Comments »
Billy’s Twitter
By Billy | May 2, 2008
If any of you use Twitter, I’ve opened an account, Billysblog. I’ll be posting interesting links related to optimization, marketing and Widemile, as well as any thoughts I have about online marketing. I’ve included a feed on the left nav bar of this blog too.
Topics: Site News | No Comments »
3 difficult optimization results and what you can learn from them (3 of 3)
By Billy | April 30, 2008
Note: This is the third post of a 3 part series, each focusing on one type of test result that is tough to deal with. Read the other 2 articles on highly mixed data and the original page beating the new variations.
Ready for the toughest of all test results? I brought in Widemile’s Chief Scientist, Vladimir Brayman, for this post to help me with some of the concepts around this topic. The last of the three results is when the results just won’t stabilize.
How does this happen?
As long as you have homogeneous traffic and enough time, a test should stabilize. Unfortunately, this is not always possible and I don’t know anyone with unlimited time. The most obvious way this occurs is when a test is designed too large, meaning you don’t have enough conversion traffic for the number of variations you are trying to test.
Additionally, getting homogenous traffic is not always easy. If your sources are too different, you can have problems. Text, banner, e-mail ads and even Yahoo vs Google traffic may behave differently. The worst case is when these sources of traffic are added mid-test. I have had tests where an e-mail campaign was done at the end of a test without my knowledge (until I asked about the huge spike in traffic!)
You can’t control all traffic coming to your page from some sources like PR, blogs, seasonal events and news. This goes back to part 1, about highly mixed data; everything there applies to this case too.
A test also may not stabilize because the test is designed with elements that are too similar. The same thing can happen when 2 elements are different but have approximately the same amount of impact. In these situations, your data will go back and forth on which of them are the winners.
Anything outside of your page that has a large influence can destabilize your test, this includes pieces of your funnel. One symptom of this is when your clickthroughs are fairly consistent but the full conversions are not. If you are testing a landing page and the sign-up process after it is very kludgey and difficult for users then it can have a large impact on your tests’ ability to stabilize. This is especially true if the experience for visitors changes. An example of this is visitors bailing from a purchase funnel because shipping to their area is prohibitively more expensive than other areas. Although they would have converted if shipping was within the average price range, they ended up not converting because of something encountered outside of the landing page, skewing your results. This is in almost every test, but the magnitude of its impact depends on what exactly occurs.
What can you do to prevent this?
If you are using a testing tool different from what you normally track your conversions with, make sure you run a baseline test so that you can compare the numbers your testing tool gives you with the ones your conversion analytics produces. They should be within about 10%-15% of each other over about a week or so. Finding a large discrepancy here will save you from headaches down the line. This essentially double checks the expected traffic numbers by ensuring you are measuring your current conversion correctly, which allows you to design a test of the appropriate size. By size, I mean ensure that you have enough testing time and within that time you will get enough traffic.
While easier said than done, it is important to look for new traffic that may be driven to your page and to segment it out. Since this shares some of the same problems as highly mixed data, those solutions apply here too.
What can you do if this happens?
First, don’t cut your tests short unless you think more data won’t solve the problem. If you don’t reach stabilization, you are wasting all the time you tested since you have inconclusive data. Always try to be as conservative as possible and end tests only when you are very confident that the test is stabilized or that there is no other choice.
Think about restarting the test if it isn’t stable. Use a smaller design. Pick the important factors (pieces) and the levels (variations) that you think will perform and are drastically different from each other. This prevents elements from looking unstable as they flip flop as the optimal.
If your only problem is that 2 variations are vying for the winning position, then they likely perform about the same. It probably is not really worth your time to wait for them to stabilize and so stopping the test and going with either of them likely will have little difference to your conversion rates.
The problem of outside funnel influence is a bit harder, but not impossible to solve. The best solution is to segment the users that are determined to be unqualified. For example, if you only ship or work with US customers and businesses, then filter out any users that are outside of the US and do your analysis from there. This can be done either at the data level if you can tell where the data came from, otherwise this can be done with a splitter or qualification page that leads people into the appropriate funnel first. This may impact your overall conversions itself though, so careful testing around these methods should be done as well.
From my experience, the problems I’ve listed in these three posts are either preventable or unlikely to occur. The value of having an optimization expert is because they can avoid these situations or at the very least extract useful lessons when they do happen. Having said that, don’t be scared to test. Once you get the hang of it, it is a lot of fun and one of the keys to effectively growing and maturing your online marketing campaigns.
CC photo credit #1: ryaninc - CC photo credit #2: jurvetson
Topics: Methodology, Testing Concerns, Testing Techniques | No Comments »
Google Web Optimizer officially launched, no AdWords required
By Billy | April 18, 2008
I just got news that Google Web Optimizer is out of beta. In addition, it doesn’t even require an AdWords account to use it anymore. This is great news for the testing industry and for all online marketers. Check it out here. In addition, there now is a dedicated Official Google Web Optimizer blog.
I’ll see if I can get some tests running just to see what the isolated tool looks like versus the integrated one. They also upgraded the setup of multivariate tests for all versions.
On another note, it’s good to see that Google saying things like “it’s hard to find a serious advertiser who doesn’t at least plan to do content testing this year.” They even mention some best practices that I’ve talked about at this blog:
- “don’t be shy: big changes generally yield big differences in performance”
- “We recommend letting your experiments run for at least two weeks, no matter how much traffic you get and how strong the results initially appear, just so the data has enough time to normalize.” - I recommended the same things in my Multivariate Testing Primer.
Also there’s a forum for Google Web Optimizer users, which isn’t new, but expect it to grow quickly with this latest announcement.
If you’re waiting for the last post in my 3 part series about difficult test results, I apologize. I’ve been sick all week and wanted to go over my last post with Vladimir Brayman, Widemile’s chief scientist, before I posted it for the world to see. It’s a very important topic and a challenging one too. I’ll try to get it out next week for sure.
Topics: Industry News | No Comments »
Great resource for landing page optimization
By Billy | April 11, 2008
I just received a link to an amazing resource from MarketingExperiments, it’s a compilation of great webinar summaries and case studies that they have done. They cover topics from landing page optimization to price testing to PPC and more. While not everything is about testing specifically, all their advice and ideas can be tested, which is why I think you all will find it valuable.
All testing should be carefully designed; it should be focused on best practices and tactics that are predicted to connect with the audience. You should take risks when testing, but they should be calculated risks.
Check it out and soak up some knowledge on optimization and get ideas to test on your site.
Topics: Landing Page Optimization | No Comments »
3 difficult optimization results and what you can learn from them (2 of 3)
By Billy | April 10, 2008
Note: This is the second post of a 3 part series, each focusing on one type of test result that is tough to deal with. Read the first article on highly mixed data.
As an optimization analyst, this is probably the hardest result to bring to a client. Oddly enough, it actually is favorable to part 1’s highly mixed data and part 3. I am talking about optimization that determines that the original page is better than the tested variations.
How does this happen?
Sometimes a page just gets it right. How would you change Google? I looked for a few variations and came across one by Andy Rutledge and another by Valacar. They both are beautiful designs and a lot of thought were put into them, but at the same time, would they really make Google more profitable? It’s definitely a tough sell and there is a big challenge in improving this type of page.
The goal is for users to search. Yes, they want users to click on ads eventually, but there’s not a whole lot they can do for ad clicks on the homepage. The best they can do is get users to search as fast as possible. So would a redesign make it more usable and readable? Maybe. To a level that it would increase their revenues? That’s tough to say.
The more simple the goals of the page, the less information and messaging the users needs, the more likely that the page will be difficult to optimize.
What can you do to prevent this?
Be careful when choosing a page to test. Find a page where the user will take some time to look at what is going on. This is another reason why most landing pages are great places to optimize, because users naturally need to be introduced to the product and sold on why to convert.
The logical thing to do would be to simply refrain from testing pages that seem to be performing well, but this is rarely a good rule. Unless it is performing well because of a lot of testing, then you don’t really know if a page is performing well or not (see my post on conversion rates.) Testing always brings surprises and personal judgment is no replacement for a test; a good looking page can perform poorly and a page with subpar creative can perform great.
What can you do if this happens?
Because of the above reasons, you may actually plan for this scenario to occur. Many people believe redesigning an old page will provide improvement, but what if it is old and performing well? In that case, you may plan to try to improve but not expect to beat the old version.
In any case, if your original page wins, then you have confirmation of your page’s success. It is unlikely that all possible improvements were tested in one test run though, so it may take a few more runs to really confirm its solidarity, but the page has won against the initial best ideas and that is an achievement.
This lesson tells you that you can move on and that is progress in itself.
Moving forward, I would try drastically different approaches, either in layout or design and testing around offers. Otherwise, I would apply the successful original page to tests for other areas of your site.
I have to be honest when I say that this rarely ever happens. Almost every page has room for improvement at every step of the conversion funnel.
Whew, I will try to get the third and toughest optimization result next week.
CC photo credit: philosophygeek
Topics: Landing Page Optimization, Methodology, Testing Concerns, Testing Techniques | No Comments »
3 difficult optimization results and what you can learn from them (1 of 3)
By Billy | March 31, 2008
Note: This is the first post of a 3 part series, each focusing on one type of test result that is tough to deal with.
There are 3 types of optimization results that people never look forward to getting.
Unfortunately, anyone who runs enough tests will run into these situations. In the following 3 posts I will go over the 3 situations and outline how they happen and how to prevent and handle them.
This first post is about tests with highly mixed data.
How does this happen?
Typically mixed test data occurs when events out of your control, or you forget to control, impact your test and skew your test results.
Your average traffic finds your tested page either through search, browsing around your website or through your planned advertising campaigns. However they get to the page, tests are designed (or should be designed!) based on how visitors will get there.
The problem arises when, outside of the scope of those involved in testing or because of some oversight, a new type of traffic is driven to the page without any preparation being done for that traffic. While traffic is good for your sample size, it is bad because those new visitors are coming in with totally different motivations, assumptions and knowledge. This means they probably will react differently to your tested elements than the traffic you were driving to it originally.
Most often this happens to me because of a new marketing push, such as an e-mail blast, new display ads or promotions at a trade show. This can happen even more unexpectedly if some outside party drives a lot of traffic to your page. A news story or blog review that innocently links to your page, can suddenly becomes a big source of new traffic.
What can you do to prevent this?
The first step is to spread the knowledge around your company that this testing is going on and that anything that may impact the page and its visitors should be run by the optimization team first.
Next, always segment out significant traffic and track it separately. If you segment, the worst case scenario is that they perform the same as your current traffic and you did a little extra work. The alternative is having to trudge through your data, trying to separate the 2 types of traffic and possibly having to restart the test if you can’t separate them out.
Lastly, be aware. Watch your data and look for big changes. If you see something strange or a sudden shift, try to find a cause. It usually will be nothing, but if you do find something, quickly build a segment for it. Even if the “new” traffic has already started hitting your page, a segment should be setup as soon as possible.
What can you do if this happens?
I would still try to segment the data in any way possible. Even taking certain days/time out of your data, may be enough to salvage your results. Do your analysis with and without those days and see if the optimal page changes. You should take extreme care when doing this still though and make sure you have statistically relevant results.
My next solution is just to restart the test. Testing is about continual growth and you can’t always get what you want out of every test. Be happy that you got some extra traffic and try to take precautions to take it into account, or prevent it, the next time around.
Let me know if you’ve ever run into these problems before and how you handled them. Look out for part 2 of this series in the next week.
Topics: Testing Concerns | No Comments »
3 parts to picking a test page
By Billy | March 26, 2008
Alright, so you’re ready to test. You’ve got tools and the skills to design a test. But when optimization begins, where should you start? While we all would love ROI to be the only driving factor in optimization, your resources and reach usually dictate what you end up testing, as well as ROI.
Here’s what I think about when looking for candidate test pages:
- Importance: Is this the best page to accomplish your goals? Take a look at your overall marketing campaign and see how important this is to the whole process. Look at drop-off points and find pages that are important but weak links.
- Technical: Will this page be easy to optimize? How much technical involvement will it require? If you’re testing dynamic elements you may need some additional help. Maybe you can test a page outside of the development schedule or on a separate server. Look for pages that have less restrictions and can be modified quickly. Also examine the page for what can’t be tested and what you may want to test on a page. Some tests are harder to create than others, both technically and creatively.
- Goals: What are you optimizing for? Pages with one goal are easier to optimize since you can drive everything on the page towards achieving that one goal. If you think a page is under performing, then it may be an easier page to optimize also. Lastly, think about how easy that goal will be to measure. If there are multiple conversion possibilities or the conversions are offline, it will be difficult to test.
In the end you are asking a multi-part question: Will a lift here be more valuable than a smaller/same/larger lift elsewhere that will take X amount of work and time?
Just remember, you can always test a page later on, even if it may not be the best candidate now. If you have the resources you can test pages simultaneously too. Just make sure they don’t impact each other in any way, so as to not skew your results.
As your optimize more and more, it will be harder to choose, but that’s a good sign. It means you’ve got a lot of great pages and that is what you want testing to do for you.
Topics: Methodology | No Comments »
1 quick but powerful test design tip
By Billy | March 20, 2008
I was going over my testing plans with my boss, Frans Keylard, today and he reminded me of a very powerful rule.
Test if something works before you try variations of it.
In this case, I was testing out two testimonials. They were quite different in the messaging, however, do I even know if testimonials are read or impact visitors at all? If I test a testimonial and no testimonial, I will immediately know if I should continue trying testimonials. If testimonials win or compare favorably against having no testimonial, then I know to test additional testimonials.
Not that I have never tested factors on/off or tried totally different factors, e.g. a testimonial against a product shot. I had a strong feeling testimonials were going to work, so I assumed they would, although I know I shouldn’t assume anything. An honest mistake, but a good reminder.
Ideally I would be testing variations, along with showing nothing, or “off”, as a variation, however in this case the page didn’t get much traffic so I was limiting my testing to the most important variations and factors.
There might be some fringe cases where this isn’t necessarily true, but in most cases you should just save extra variations for future runs and first find out if your factor has any impact on the page. Maybe I need to read some of my old posts more often.
Topics: Methodology | No Comments »
Looking forward to 2008
By Billy | March 19, 2008
AdAge asked everyone on their Power 150 blog list to contribute a short snippet about “what technology marketers should be paying most attention to in 2008.” I got included in the list and found it a good way to introduce someone to testing. If you have any marketing colleagues that still aren’t sure why testing is valuable, here is my full, unedited submission:
Site optimization is already becoming a focus of every marketer in 2008. Why? Because testing and tuning websites is a natural extension to the proliferation of web analytics. Marketers know how important analytics are to their campaigns, but even with all the valuable data gathering tools out there, there is not a straightforward next step to improve websites. Site optimization through multivariate and split testing helps turn analytics into action, allowing websites to improve and grow with their audience.
In addition, the rising costs of PPC makes each click even more important. Using site optimization, the large gains that come from optimizing SEM are now arriving at the post-click stage. These gains are not isolated to SEM either; site optimization improves ROI across the board. As the market matures, the need for site optimization will continue to grow, making it essential that every marketer considers it in 2008.
Topics: Why Test? | No Comments »