Tips for A/B testing?

Question

There seems to an increasing number of A/B testing services (such as Google Website Optimiser) and people using them. Have you got any tips on using and analysing the results from them?

Answer 1

There's a really nice article by Cennydd Bowles that you'll probably find useful Statistical significance & other A/B test pitfalls

Answer 2

Philip

Re: Analysing the results. Take a look at the Easy statistics for AdWords A/B testing, and hamsters from Jason Cohen.

It's one of the best articles i've ever read about understanding the statistical relevance of the results of an A/B test. Plus (and this is a big plus for me) it's fun to read.

The main message:

The way you determine whether an A/B test shows a statistically significant difference is:

Define N as "the number of trials." - (Click results for variant A + Variant B)

Define D as "half the difference between the 'winner' and the 'loser'."

The test result is statistically significant if D² is bigger than N.

A full explanation is provided in the article.

All the best,

Matt

Answer 3

I've been involved with Website Optimizer and Omniture, Marketo as well as some home-grown A/B testing tools. My key finding is this: Garbage In - Garbage Out.

The key question is "What are you testing? WHo is deciding that?" Is it just "Let's try and beat the control" or is it "Lets learn about our audience." The latter is much more important. You have to get to the WHY questions, not just the WHICH.

At Intuit, as example, they tested all kinds of terrible designs by committee and never really moved the needle. They also never learned why one thing was better than another.

User Experience has to be about getting inside people's heads. It can't just be randomly throwing pixels together and hoping it is better than the last effort.

Sorry, maybe I flew off-topic. I have seen AB testing go awry in many ways. And don't get me started on multivariate testing (DOE).

Last thought: Avinash Kaushik's blog and books are awesome to learn about analytics. He is a superstar in the field.

Answer 4

There is a pretty good article by Jon Von Tividar at http://www.futurenowinc.com/resources/abtesting.pdf

A good bit about avoidance of false maxima - I am with glen and think you can only possibly reach the peaks by applying a whole host of techniques first - then tweak using A/B or Multivariate testing. In web design at least you do need an 'intelligent creator' and cannot leave it up to evolution by a/b testing natural selection.

[NB: I am also amazed how often I see people claim results with abslolutely no understanding of even basic stats and significance - and don't get me started on marketing people doing creative concept testing...]

Jon

Answer 5

You might find Smashing Magazine's The Ultimate Guide To A/B Testing a useful A/B testing resource:

It includes a section on Tools For A/B Testing, as well as covering the following:

• What Is A/B Testing?
• What To Test?
• Create Your First A/B Test
• Do’s And Don’ts
• Classic A/B Testing Case Studies
• Resources For Deep-Diving Into A/B Testing