A/B Usability Testing Technique Format?

Question

I'm currently preparing a usability test for a module interface I'm designing and I want to be able to structure the test to ensure the best results.

Currently, there are 3 versions of the interface, and three groups to be tested:

Version A

• Module 1
• Module 2
• Module 3

Version B

• Module 1
• Module 2
• Module 3

Version C

• Module 1
• Module 2
• Module 3

Currently, I'm a bit unsure on how to structure and randomize what each group tests and the order they test each Version to ensure people don't "learn" how to do the tasks I'll ask them to do.

My idea is this so far for the test:

Group A: Version A1, Version B2, Version C3

Group B: Version A2, Version B3, Version C1

Group C: Version A3, Version B1, Version C2

Would it make a difference if different overall version was tested first for different groups (ie. Group B would do Version B3 first, then Version C1, then Version A2) and does the module connected to the version tested make a difference?

Thanks and would appreciate any feedback.

Answer 1

You need to define what "best" means for your interface, since you appear to be combining different methodologies and might not find the answers you are looking for.

An A/B test is a quantitative method where you decide on a particular metric (like button clicks) and you see if version A or B yields more. You divide a large number of users into two groups and show a single experience to each group and measure which one has the larger impact on your metric. You will learn what works but you won't learn why. For example, will a page with puppies or kittens on it sell more shoes?

A usability test is a qualitative method where you explore different ways that users interact with an interface to see if there are potential problems. You take a small number of users and run them through a typical scenario and observe as they interact with the interface. You will learn if there are issues, and often learn why they happened. For example, can users add shoes to their wishlist in our new prototype?

A balanced comparison is where you are trying to see if there is a preference for one version, but trying to control for the positioning bias by randomizing the order. These are usually more quantitative than qualitative since you are trying to rank things, not just get feedback on each one. For example, on a scale of 1-10, how cute is each puppy?

If you are trying to optimize something like time on task using all the modules, that can be done with an A/B test, though you would need an audience large enough to spread across 27 combinations to test to see if any resulted in a faster time. Or you could do many tests of fewer combinations but would eventually have to pit the winner of each test against each other.

If you are trying to learn if there are potential usability problems in each module, it's probably best to test each one individually unless they are connected in some way. If each module is for a specific task, build a scenario where a user has to do three things, one from each module type. It won't really matter which version of each module they use, but keep them the same for simplicity's sake. You might consider randomizing the order of the tasks/modules if you are concerned that later tasks are only possible because of practice, but that's why testing each module separately might be better.