A/B testing in user test

Question

I designed 4 versions of a checkout page. I want to make some user test to know which version performs better. Do you think that I have to do the tests with the same person? the big change in the interface is the layout. The informations are the same, so I'm afraid to influence the users, since they already know how the system works (after seeing it once). Do you think that is simply better asking, which do you prefer among these 4? or I have to test the entire checkout process to see if new layouts show some problems?

Answer 1

Should each user try all four versions?

Re-using the same user gives you the ability to statistically discriminate smaller performance differences among your versions, assuming you know the right statistics to apply (e.g., repeated-measures ANOVA, for task completion time). Re-using users allows you to factor out the users’ idiosyncratic differences (as indicated by the user’s average performance across all four versions), so you can focus on the remaining variability that is attributable to the differences in the versions. Re-using users can also cut costs associated with the overhead of recruiting users –in your case, to get the same amount of data, you only need to recruit one quarter of the number of users you would need if each user only tried one version.

It’s legitimate to be concerned that exposure to the first version will impact performance on later versions for each user. The solution is to “counter-balance” the order of versions so different users get a different sequences of versions and the average serial position of the versions is the same across users. Ideally, you run exactly 24 users and each user gets one of the possible 24 (factorial(4)) sequences of versions. At the very least, the number of users should be an exact multiple of 4, and one quarter of the users should get have a sequence of versions of 1-2-3-4, another quarter should have 2-3-4-1, another should have 3-4-1-2, and another should have 4-1-2-3. Mix up the sequences randomly when you dole them out to users (e.g., first user gets 2-3-4-1, second gets 3-4-1-2, third gets 1-2-3-4, and fourth user gets 4-1-2-3).

Do I ask users which they prefer?

Subjective preference is only moderately correlated with objective user performance. If you want to know which version performs best objectively (e.g., least errors, minimum task time), then use an objective measure. Which version excited the users the most (or, at least, annoyed them the least) is a legitimate UX question, but different than which version was, in fact, easiest to use. You can, of course, measure both objective performance and ask users to rate the versions subjectively to capture the “full” user experience.

In either case, subjective preference may be affected by the sequence of versions you present the users, much like objective performance. Users may tend to like the first thing they see. Or maybe they like the last thing they see. I don’t know. But, more to the point, who cares? To get assured accurate averages, counter-balance the order of the versions when asking users which they prefer –just like you would for objective measures.

Do I test the entire checkout process?

Statistically, you can use a smaller sample size by focusing the task on just the things your versions vary on. This reduces the random effects from other things adding “noise” to your data that makes it harder to hear the “signal.” Thus, only do as much of the task that should be affected by your different layouts. For example, if the place to enter the credit card number is the same for all versions, stop before you get to that part.

On the other hand, don’t make the task be unrealistic (e.g., don’t interrupt the user’s flow and concentration to tell them to skip this step or that step). Also, this usability test could be an opportunity to gather data on the page in general. Maybe you can find and fix issues beyond any associated with the layout. If you have to go through a lot of trouble to recruit your users, it would be a shame to “waste” them.

Answer 2

The main problem you'll run into if you just plainly ask “which do you prefer” is that it usually turns out to be preference based on fashion not function. The reason we do test, especially usability test, is to determine which one performs best. A test we ran that seemed to bring good results is a usability test where we had user’s test on all of the interfaces, we had three, and we have them run through the same scenarios and we watch. If your interfaces have big enough differences it will impact the results of the test. You just need to determine if they do. For example: Are your interfaces just an apple in different colors (that’s when you ask which do you like more) or are your interfaces an apple, cat, car, and bazooka (then you run a usability test and then you can ask them their thoughts). But from my experience we ran test with 8 -12 people on 3 different interfaces that all did the same function. It turned out well. I hope this helps.

Answer 3

TL;DR Measure real-world impact (in terms of revenue, not conversion rates) using a true split test, not user preferences.

Showing all versions to a real-world user will likely result in confusion and bump up abandonment rates, regardless of the order in which you show the variations.

The way split testing works is this: come up with some variations, then split your audience and show one variation to each audience member. So, you should only be showing a given visitor version A, B, C, or D, even if they come back multiple times; it should be the same for each visit by that person, unless and until you change the design permanently.

Now, you mentioned "user testing" to figure out which variation performs best. If you have a controlled environment and can test how successfully participants complete an action for each variation, okay. But in the real world, you don't want to confuse the visitor by having a page look different every time they visit. Besides, you can't even know that a real-world user will visit that page 4 times.

And as others are pointing out, testing preferences or performance (in terms of completion) in a lab doesn't tell you anything about how a variation performs in the wild. A lab user who is told, "Here, try to checkout on this page," will try to checkout, period, because that's why they are there. A real-world user, presented with the same page, may well abandon the effort. Your lab tests will not tell you anything about how likely it is that the real-world user gives up.

Answer 4

I designed 4 versions of a checkout page.

Sounds like A/B/C/D testing :)

I want to make some user test to know which version performs better. Do you think that I have to do the tests with the same person?

I do, because if you look at the definition of A/B testing, it requires that there is a comparison between different versions of the same design to see which one statistically performs better.

The informations are the same, so I'm afraid to influence the users, since they already know how the system works (after seeing it once). Do you think that is simply better asking, which do you prefer among these 4?

How many users do you plan on including in the sample size? If you are doing random assignment for the testing like A/B usually requires, then different users will be introduced to different variations of the design to offset the bias.

I have to test the entire checkout process to see if new layouts show some problems?

What problems are you looking to resolve in the checkout process? You may not want to use A/B testing then, since A/B testing is generally not used to point out larger problems in the system, such as credibility or trustworthiness or help you understand why a user liked one design over another. It is primarily used to supplement other qualitative methods that help you further understand what your users want.

Answer 5

This is exactly what UsabilityHub was designed for. Below are some excerpts from their home page:

---