I've read that A/B testing and MVT are best for testing smaller scale changes (i.e. button color, headline copy, etc), but are they also suitable for testing two (or more) very different designs? What's the best practice in a situation like that?

  • What exactly are you trying to test? i.e. what are you trying to get measurable results against? (which button gives you more sales? Which headline encourages more people to read a page?) Set your measurement first then you can decide how to test it.
    – JonW
    Nov 6, 2012 at 16:28
  • I create website templates for small local businesses, so my ultimate goal is to find out if one design converts better than another. Conversions here being defined as a phone call or a contact form completion. So I'm not even trying to test a particular color or copy -- only the layout (placement of elements).
    – ccnokes
    Nov 6, 2012 at 16:36
  • 2
    Unless the number of definitively attributable responses are in a large enough number, then you may need to be very careful about how you react to your 'findings' as it's very easy to misunderstand why events occur. Nov 6, 2012 at 16:53

1 Answer 1


AB testing is perfect for a situation in which you can relate the difference in responses to the thing that is different between the two versions.

With two very different versions you can only gauge an overall response but not attribute a better or worse response down to a particular detail, and that makes it harder to put forward a reasoned argument for ongoing design. You don't know which features should be kept or should consider removing at a later date for example.

So you need to ask yourself:

  • what am I comparing against what
  • how am I going to measure the response
  • how much feedback do I need to make it worthwhile
  • how can I relate these metrics to the design in question
  • what am I going to do when I have the results
  • how will I explain the results to others

A good exercise is to walk through what will happen if your testing shows A to be the preferred version - and then do the same if B were to be preferred. Also consider what if there were no preference. This will help you determine what is important and to perhaps reconsider how your testing is done


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.