User Experience Stack Exchange is a question and answer site for user experience researchers and experts. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

There seems to an increasing number of A/B testing services (such as Google Website Optimiser) and people using them. Have you got any tips on using and analysing the results from them?

share|improve this question


Re: Analysing the results. Take a look at the Easy statistics for AdWords A/B testing, and hamsters from Jason Cohen.

It's one of the best articles i've ever read about understanding the statistical relevance of the results of an A/B test. Plus (and this is a big plus for me) it's fun to read.

The main message:

The way you determine whether an A/B test shows a statistically significant difference is:

Define N as "the number of trials." - (Click results for variant A + Variant B)

Define D as "half the difference between the 'winner' and the 'loser'."

The test result is statistically significant if D2 is bigger than N.

A full explanation is provided in the article.

All the best,


share|improve this answer
Brilliant, thanks! – Zoltán Gócza Jan 15 '10 at 12:32

There's a really nice article by Cennydd Bowles that you'll probably find useful Statistical significance & other A/B test pitfalls

share|improve this answer

You might find Smashing Magazine's The Ultimate Guide To A/B Testing a useful A/B testing resource:

It includes a section on Tools For A/B Testing, as well as covering the following:

  • What Is A/B Testing?
  • What To Test?
  • Create Your First A/B Test
  • Do’s And Don’ts
  • Classic A/B Testing Case Studies
  • Resources For Deep-Diving Into A/B Testing
share|improve this answer
Smashing's article does appear to be very well researched and educational +1 – jffgrdnr Jun 24 '10 at 19:16

I've been involved with Website Optimizer and Omniture, Marketo as well as some home-grown A/B testing tools. My key finding is this: Garbage In - Garbage Out.

The key question is "What are you testing? WHo is deciding that?" Is it just "Let's try and beat the control" or is it "Lets learn about our audience." The latter is much more important. You have to get to the WHY questions, not just the WHICH.

At Intuit, as example, they tested all kinds of terrible designs by committee and never really moved the needle. They also never learned why one thing was better than another.

User Experience has to be about getting inside people's heads. It can't just be randomly throwing pixels together and hoping it is better than the last effort.

Sorry, maybe I flew off-topic. I have seen AB testing go awry in many ways. And don't get me started on multivariate testing (DOE).

Last thought: Avinash Kaushik's blog and books are awesome to learn about analytics. He is a superstar in the field.

share|improve this answer
Excellent point Glen - Tools are only as good as the people who use them! – Matt Goddard Jan 15 '10 at 17:13

There is a pretty good article by Jon Von Tividar at

A good bit about avoidance of false maxima - I am with glen and think you can only possibly reach the peaks by applying a whole host of techniques first - then tweak using A/B or Multivariate testing. In web design at least you do need an 'intelligent creator' and cannot leave it up to evolution by a/b testing natural selection.

[NB: I am also amazed how often I see people claim results with abslolutely no understanding of even basic stats and significance - and don't get me started on marketing people doing creative concept testing...]


share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.