I just stumbled upon Google Website Optimizer. Is this the way to go?
How do you normaly do A/B tests with your website?
I used to work at Intuit and they tested everything. Here are some lessons I learned:
Also, there are other kinds of tests. Satisfaction surveys being one example. But a website has lots of goals besides those that can be measured on Google. For instance, Ask people who visit the site: "What does this company do?" And see what they say. Test different messaging. Which gives the most accurate rendition.
Bottom line: (imho) A/B testing is only useful in high traffic controlled situations on e-commerce (or highly "goal oriented") sites. A standard website, you are better off with hallway usability tests.
For a quick intro there's a good video tutorial on YouTube entitled "Planning and Running your First Experiment with Google Website Optimizer". It's over 1h long but well worth your time if you're new to GWE and I think it answers your question.
GWE is focused on testing for improved conversion rates. Would be great to know your opinions if increased conversion is always related to better usability.
I'd say Google Website Optimizer is definitely the way to go. It's good to learn as much as possible about A/B testing so you know what to test and how. But as far as tools go, GWO is a great one.
I'd recommend reading Always be testing (Amazon.com) which is especially written for Google Website Optimzer. It is aimed at beginners and does a very good job of introducing the subject and giving ideas to begin testing with.