It’s been a fun journey here at Widemile and although Widemile, the name, is disappearing, our mission to bring optimization to the forefront of online marketing has not ended. I will be moving this blog over to the Webtrends domain very soon, for now, please see my post on the Webtrends blog for my thoughts on the acquisition.
Thanks for being a loyal reader of my blog, I will be posting a lot more once I switch over to Webtrends, so don’t forget to check it out.
-Billy
Webtrends is hosting a completely transparent marketing campaign, called The Open Campaign. In the campaign, Webtrends and a few partners, including Widemile, are conducting their various specialties and sharing the results with the public. The idea is to let you all learn about all the great marketing technologies out there and how it ties together with Webtrends, as well as seeing the performance of the campaign itself.
Check out the Site Optimization section where I will be discussing the test we designed for The Open Campaign and the results. See for yourself exactly what its like to do optimization with Widemile Optimize and how my brain works throughout the process.
A huge part of doing optimization well is knowing what to test (put garbage in, get garbage out), so keeping up with good design philosophy is extremely valuable. While brushing up on web design, I came across a Smashing Magazine article on UI design trends by Janko Jovanovic. He uses a lot of great examples of good design, some of which are perfect for illustrating some optimization options.
With that in mind, I’m going to examine one of the sites mentioned and discuss the good, the bad and the testing opportunities I see. The (lucky?) site I picked was Picnik, which has done a commendable job on their registration strategy. (Also, like Widemile, they are a Seattle start-up.) I only wish the site wasn’t flash based, which is more difficult to optimize. Despite that, my thoughts on test variations and best practices are still applicable to it and any other registration campaigns.
Quick summary: Picnik is an online photo editing application. You can upload photos and do easy photo editing all within the browser. You can try out the app, even exporting and saving photos, without registration.
Let’s get started by checking out their form:
Although a bit busy, I like the way the form assists users. It has a green highlight to for the selected field and dynamically pops up error messages (see the username alert below). Additionally, it hides and locks the “again” fields until there is valid input in the corresponding field.
One highlight is how this is a good example of when a lightbox/page overlay type form might be appropriate (note that behind the form is the page I was working on, which has been darkened). Why is it appropriate? Because this is the form that pops up after the user clicks “Register.” It makes sense to be direct and reduce additional marketing if the user indicates they want to sign-up by clicking directly on the register button.
Is this right for your site/landing page/microsite? It’s hard to say, but I would recommend testing it. This would fall into the category of a funnel test because it eliminates a page in the registration funnel. As long as your full page and lightbox form don’t have any glaring issues, you should quickly see the influence of whether a small and direct lightbox form works, or if a whole page with additional information is necessary.
In terms of testing this overlay form, there are a few big opportunities for improvement.
So how does Picnik capture users that don’t click register directly? They offer it after a photo is saved:
As you can see, this page has a lot more content than the lightbox form since its a full page. It has the job of pushing someone into registering after having used the product. This is a good technique (mentioned in Javanovic’s article), but there’s always the question of if you’re offering too much or too little. Testing how much to offer would be a very interesting and fruitful optimization campaign.
Overall, I’m not a huge fan of this page, but I do like the approach. It has continuity at the top, showing the actual photo edited, and the form and main registration benefit (”Want Picnik to keep a copy?”) are prominent. Also, they have structured the page to prioritize their conversion goals, keeping the focus on registration but still advertising the opportunity for people to print their photos or sign up for premium service below.
Here are a few recommendations to improve this page:
Optimizing for registration involves many steps, beyond just improving the registration pages. You can delve into when to ask for registration, test the ROI of emphasizing different products and then executing segmentation focused pages as well. However the easiest returns will come from some simple fixes like I’ve discussed above.
I hope this was helfpul talking over a real example, let me know if you’d like me to do more of these and if there’s any great sites out there I should look at.
If you missed it, see Part 1 (A/B Split Testing) and Part 2 (Multivariate Test Basics).
With the basics of part 2 down, it’s time to start designing a multivariate test. Every optimization project has different challenges and goals, luckily though, there are a few rules that apply to every multivariate test design. These rules fit into two categories: technical rules and content rules.
Technical rules:
These rules ensure statistical significance by constraining the test to the appropriate size at the beginning and then letting the test gather the proper amount of data at the end.
Running a test full factorial, if your traffic supports it, may be a good choice if you’re testing content that you believe to have many interactions or if you only want to test 2 factors with 2 levels each. (Note: the smallest fractional factorial test size is 3 factors with 2 levels each.) Typically though, you’ll want to run a fractional factorial test to save time and expand the number of factors and levels you can test.
In order to find out how many factors and levels you can test, you need to have some idea of your predicted page views, conversions, as well as an estimate of lift. The reason that lift matters, is that a large lift will get you more conversions and so your test will stabilize quicker. Because of this, I would be conservative with lift estimates to ensure that the test is not designed too large. At Widemile, we have a large list of arrays available to our tool and have calculated the approximate conversions needed to stabilize, allowing me to look at the three criteria I listed and find the arrays that are statistically viable for testing. You should look for something similar with your tool of choice.
To figure out when a test is stabilized, I prefer to primarily look at level influence stabilization with experiment conversion rate stabilization for support. Widemile Optimize shows this using graphs, so I simply look for horizontal trending of lines, meaning winning levels and experiments stay winners and their level of influence or conversion rates stay fairly constant (look horizontal) over 3-5 days. If you don’t have graphs available, the historical cumulative conversion rate for your experiments and see if there is a lot of variance between the latest few days of your test.
Content rules:
The content rules are closely tied together. In effect, they ensure that the items selected for testing have purpose and that they don’t needlessly expand the size of your test, reducing its efficiency. I begin designing tests by creating hypothesis regarding issues with the page and then choose factors and design levels to address those issues.
An example hypothesis is “Having a hero shot on the right side of the page causes users to ignore the important value proposition on the left side.” To test this, I would choose hero shot position as a factor and then have “left side hero shot” as the baseline level and “right side hero shot” as the second level. This example also illustrates that, other than headlines and images, testing layout is possible with creative use of CSS and sometimes JavaScript. As long as you can revert from one to another and it matches the other factors and levels, you are at liberty to test anything.
Coming back to the rules, make sure that you are testing as few items as possible to find out what you need. Before testing a collection of lifestyle hero shots, choose one and test it against an iconic hero shot. This will save you the time of going down a path of testing something that may not work.
Lastly, you aren’t going to be able to get the best page on the first run or even second, third, etc. If you knew what your audience liked 100% of the time then you wouldn’t need testing. Remember to think of your overall test plan beyond just the first run, so that you can answer all the questions you need without having to force everything into one test.
In summary, determine what you’re trying to achieve, select the proper testing method to meet those goals and then make sure to be purposeful and efficient with the content you end up testing in front of your visitors. Testing and optimization is not difficult, although it can be tough to start. Follow these rules and you’ll be on your way to conquering conversion rates, bounce rates, funnel drop-offs and many other metrics.
Photo credit: Aranda\Lasch (CC)
I recently read Douglas Bowman’s blog, Google’s former Visual Design Lead, about why he left Google. In it, he describes how the engineering culture contributed to his decision to leave:
When a company is filled with engineers, it turns to engineering to solve problems. Reduce each decision to a simple logic problem. Remove all subjectivity and just look at the data. [...] that data eventually becomes a crutch for every decision, paralyzing the company and preventing it from making any daring design decisions.
He then references Google testing 41 shades of blue and a recent debate he had over “whether a border should be 3, 4 or 5 pixels wide” and was asked to provide data to back up that decision.
Bowman’s post brought up some feelings of disappointment towards Google because despite having their own optimization tool, they did not create a culture to encourage their lead designer to expand his work and actually drove him away. Optimization and testing is still in its early stage, so mistakes will be common, however I hope news like this doesn’t scare others away from testing.
Rather, I hope companies can learn from Bowman’s experience. Instead of holding designers to every detail, testing should allow them to explore, learn and refine their ideas. Testing should not prevent “any daring design decisions,” I feel it should actually encourage them. As I said before, gamble with your conversions to raise them.
In the end, it’s all about having an understanding of how testing should and should not be used. You can use testing to find the best shade of blue, but that doesn’t necessarily mean that’s what you should be testing right now. Don’t be afraid to take a step back and try something new rather than fiddling with details, testing tools give you that freedom. Big risks, reap big rewards in optimization. Not taking risks leads to inefficient testing and, in Google’s case, a designer’s resignation.
