Tell me more ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.
up vote 1 down vote favorite

I've read that A/B testing and MVT are best for testing smaller scale changes (i.e. button color, headline copy, etc), but are they also suitable for testing two (or more) very different designs? What's the best practice in a situation like that?

share|improve this question
What exactly are you trying to test? i.e. what are you trying to get measurable results against? (which button gives you more sales? Which headline encourages more people to read a page?) Set your measurement first then you can decide how to test it. – JonW♦ Nov 6 '12 at 16:28
I create website templates for small local businesses, so my ultimate goal is to find out if one design converts better than another. Conversions here being defined as a phone call or a contact form completion. So I'm not even trying to test a particular color or copy -- only the layout (placement of elements). – ccnokes Nov 6 '12 at 16:36
2  
Unless the number of definitively attributable responses are in a large enough number, then you may need to be very careful about how you react to your 'findings' as it's very easy to misunderstand why events occur. – Roger Attrill Nov 6 '12 at 16:53

1 Answer

up vote 3 down vote accepted

AB testing is perfect for a situation in which you can relate the difference in responses to the thing that is different between the two versions.

With two very different versions you can only gauge an overall response but not attribute a better or worse response down to a particular detail, and that makes it harder to put forward a reasoned argument for ongoing design. You don't know which features should be kept or should consider removing at a later date for example.

So you need to ask yourself:

  • what am I comparing against what
  • how am I going to measure the response
  • how much feedback do I need to make it worthwhile
  • how can I relate these metrics to the design in question
  • what am I going to do when I have the results
  • how will I explain the results to others

A good exercise is to walk through what will happen if your testing shows A to be the preferred version - and then do the same if B were to be preferred. Also consider what if there were no preference. This will help you determine what is important and to perhaps reconsider how your testing is done

share|improve this answer
And here's a nice example of 37 signals A/B testing a couple of radical different design examples 37signals.com/svn/posts/… – adrianh Nov 7 '12 at 10:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.