Benchmark for usability test

Question

I'm working on a usability test of a e-commerce website. The test is a validation test (the kind of test performed close to service's release).

I read that, for these kind of tests, I have to use benchmarking to compare tests' metrics. My doubt is, what is kind of benchmark I must use ?

Would timing be one of them ? if so, do I have to use "thinking aloud" protocol? I saw that think aloud slows down users in performing given tasks, and it "forces" users to think about what they are doing (not properly a typical way of use a system, especially in a environment where there are some distractions).

What do you think about it?

What kind of benchmark metrics I must consider?

In addition, another doubt is, benchmarking compares data gathered during tests with same data gathered before test. My doubt is, benchmarking data must be taken from expert users, myself, novice users or someone else?how can I rely on these data?I don't want to fall in error, considering a good completion time (for example) one that could depend on level of expertise.

Answer 1

You don't have to use a particular user research method at the time of release. You use the one that best meets the needs of your team. It could be a benchmark usability study, it could be a longitudinal study that considers the usage of your website over time, it could be a simple usability study that looks at the most important features in this release. You should determine what your needs are, and then determine the user research methodology that best meets your needs.

If a benchmark usability study is one that meets your needs the best, and if there has never been a benchmark usability study before, then you're in the position of determining what it is that you are going to benchmark. The goal of a benchmark study is to validate that your latest version meets your user experience goals for the application. If you don't have user experience goals for your application (the application itself, not just this most recent version), then you'll need to create those first. I tend to approach a new benchmark study in this way:

1. The tasks should be ones that are the most important for the application overall, not the newest ones that have been added or changed in this release. If time allows, you can add some tasks that are specific to new/changed features in this new release.
2. The benchmark uses metrics such as the following: time on task, success rate / failure rate, number of user errors, number of system errors, satisfaction rating.
3. As a result of the above, most especially time on task, a benchmark study does not use a think aloud protocol. The user should be instructed to simply do the tasks and not talk out loud (unless that's really how they operate if they're alone and no-one is watching), except to say when they are done with the study. If you give them a written task list (for a benchmark study, I do, to minimize the amount of interaction that they have with the test moderator during a study), ask them to read the new task out loud before they begin it so that you know when they have started the task. (And yes, tasks should be in the user's language and not lead your user down a particular path. I'm not going to go into detail about that here because that's just good user research, and hopefully doesn't need to be discussed in detail here.)
4. The participants for the study are your target users for your application. If you use personas or other type of user profile, then you should recruit for people who match your profile. If you have multiple personas/profiles, I usually recruit for an even mix of personas.
5. Since this study is more quantitative in nature than some other user research methods, you likely need a larger number of participants than you need for other methods like a standard think-aloud usability study. For a benchmark, I consider 10 participants to be the absolute minimum; I've done benchmarks with as many as 50. How many participants you need is a function of the number of personas/user profiles that you have, how many different areas of the application that you will test, and how statistically significant you need your results to be.

Scott Berkun has a pretty old, but still worthwhile, essay about the art of usability benchmarking that you might find to be a useful primer.

Answer 2

"Thinking aloud" tests are more for design and prototype testing sessions, not validation sessions. For validation you want to check timing, and you want to check correctness of data: if your users are often entering incorrect data, you may need additional validation.

Yes, you need a variety of test subjects: beginners, experts, those familiar with your products, those unfamiliar with your products.

You might also want to capture your users' path. For example, if they often leave the cart while checking out, but don't buy any additional merchandise, determine why - perhaps they find that the buttons or prompts are confusing.

EDIT: add timing to the path data. Find out where your users consistently slow down. You can average the times from different users to establish a benchmark, but you can also have benchmarks from each of the categories of testers. I'm often interested in the fastest overall time, so I know what experienced users are capable of. I also want to know which screens the various users stall at. If every other screen on your site is completed in 40 seconds on average, a screen that takes 10 minutes for the median of all users to complete means something's likely very wrong.