I just stumbled upon Google Website Optimizer. Is this the way to go?

How do you normaly do A/B tests with your website?

4 Answers 4


I used to work at Intuit and they tested everything. Here are some lessons I learned:

  1. You need traffic. You don't get statistical confidence unless you have enough traffic. It's a waste of effort to test something that will take 4 months to resolve. Multivariate testing takes even more traffic.
  2. What are you testing? At Intuit, they tested horrible ideas that were designed by committee. This is part of the Garbage-In, Garbage-Out maxim. Deciding what to test is a critical part of the process and often ruined by business marketing people.
  3. What are you measuring? It is critical to define success very narrowly. A test should be a hypothesis: This metric will increase if we do XYZ. You are testing if the theory is true. Don't just throw stuff up there and then compare every metric under the sun.
  4. Seasonality. Often, it's not the A or the B. It's the season. This is why tests need to be targeted and controlled. In other words, A may be better in the summer and B in the winter. So the winner depends on the season.

Also, there are other kinds of tests. Satisfaction surveys being one example. But a website has lots of goals besides those that can be measured on Google. For instance, Ask people who visit the site: "What does this company do?" And see what they say. Test different messaging. Which gives the most accurate rendition.

Bottom line: (imho) A/B testing is only useful in high traffic controlled situations on e-commerce (or highly "goal oriented") sites. A standard website, you are better off with hallway usability tests.

  • I always admired Intuit for celebrating failure and learning from mistakes (at least by what I've read about them, ie: businessweek.com/magazine/content/06_28/b3992001.htm). Dec 3, 2009 at 19:57
  • The seasonality is a good point. And its always a good idea to keep wondering whether something in 'the environment' of the website has changed (say the kinds of users coming to the site), due to some external factor.
    – PhillipW
    Sep 7, 2010 at 9:11

For a quick intro there's a good video tutorial on YouTube entitled "Planning and Running your First Experiment with Google Website Optimizer". It's over 1h long but well worth your time if you're new to GWE and I think it answers your question.

GWE is focused on testing for improved conversion rates. Would be great to know your opinions if increased conversion is always related to better usability.

  • Somehow it's related. If you make your "sign up" button really small it's related. But I guess that's not your question. But yes, Sign-Up forms, payment forms etc. have to be thought through because they are a part and from a business point of view the most important part of the user experience you are delivering.
    – tamimat
    Dec 1, 2009 at 10:03

I'd say Google Website Optimizer is definitely the way to go. It's good to learn as much as possible about A/B testing so you know what to test and how. But as far as tools go, GWO is a great one.


I'd recommend reading Always be testing (Amazon.com) which is especially written for Google Website Optimzer. It is aimed at beginners and does a very good job of introducing the subject and giving ideas to begin testing with.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.