I'm working on a usability test of a e-commerce website. The test is a validation test (the kind of test performed close to service's release).

I read that, for these kind of tests, I have to use benchmarking to compare tests' metrics. My doubt is, what is kind of benchmark I must use ?

Would timing be one of them ? if so, do I have to use "thinking aloud" protocol? I saw that think aloud slows down users in performing given tasks, and it "forces" users to think about what they are doing (not properly a typical way of use a system, especially in a environment where there are some distractions).

What do you think about it?

What kind of benchmark metrics I must consider?

In addition, another doubt is, benchmarking compares data gathered during tests with same data gathered before test. My doubt is, benchmarking data must be taken from expert users, myself, novice users or someone else?how can I rely on these data?I don't want to fall in error, considering a good completion time (for example) one that could depend on level of expertise.

  • Clearly the benchmark is drunk testing. Bring a laptop to bars and begin the magic Mar 4, 2014 at 20:05

2 Answers 2


You don't have to use a particular user research method at the time of release. You use the one that best meets the needs of your team. It could be a benchmark usability study, it could be a longitudinal study that considers the usage of your website over time, it could be a simple usability study that looks at the most important features in this release. You should determine what your needs are, and then determine the user research methodology that best meets your needs.

If a benchmark usability study is one that meets your needs the best, and if there has never been a benchmark usability study before, then you're in the position of determining what it is that you are going to benchmark. The goal of a benchmark study is to validate that your latest version meets your user experience goals for the application. If you don't have user experience goals for your application (the application itself, not just this most recent version), then you'll need to create those first. I tend to approach a new benchmark study in this way:

  1. The tasks should be ones that are the most important for the application overall, not the newest ones that have been added or changed in this release. If time allows, you can add some tasks that are specific to new/changed features in this new release.
  2. The benchmark uses metrics such as the following: time on task, success rate / failure rate, number of user errors, number of system errors, satisfaction rating.
  3. As a result of the above, most especially time on task, a benchmark study does not use a think aloud protocol. The user should be instructed to simply do the tasks and not talk out loud (unless that's really how they operate if they're alone and no-one is watching), except to say when they are done with the study. If you give them a written task list (for a benchmark study, I do, to minimize the amount of interaction that they have with the test moderator during a study), ask them to read the new task out loud before they begin it so that you know when they have started the task. (And yes, tasks should be in the user's language and not lead your user down a particular path. I'm not going to go into detail about that here because that's just good user research, and hopefully doesn't need to be discussed in detail here.)
  4. The participants for the study are your target users for your application. If you use personas or other type of user profile, then you should recruit for people who match your profile. If you have multiple personas/profiles, I usually recruit for an even mix of personas.
  5. Since this study is more quantitative in nature than some other user research methods, you likely need a larger number of participants than you need for other methods like a standard think-aloud usability study. For a benchmark, I consider 10 participants to be the absolute minimum; I've done benchmarks with as many as 50. How many participants you need is a function of the number of personas/user profiles that you have, how many different areas of the application that you will test, and how statistically significant you need your results to be.

Scott Berkun has a pretty old, but still worthwhile, essay about the art of usability benchmarking that you might find to be a useful primer.


"Thinking aloud" tests are more for design and prototype testing sessions, not validation sessions. For validation you want to check timing, and you want to check correctness of data: if your users are often entering incorrect data, you may need additional validation.

Yes, you need a variety of test subjects: beginners, experts, those familiar with your products, those unfamiliar with your products.

You might also want to capture your users' path. For example, if they often leave the cart while checking out, but don't buy any additional merchandise, determine why - perhaps they find that the buttons or prompts are confusing.

EDIT: add timing to the path data. Find out where your users consistently slow down. You can average the times from different users to establish a benchmark, but you can also have benchmarks from each of the categories of testers. I'm often interested in the fastest overall time, so I know what experienced users are capable of. I also want to know which screens the various users stall at. If every other screen on your site is completed in 40 seconds on average, a screen that takes 10 minutes for the median of all users to complete means something's likely very wrong.

  • What do you mean for correctness of data?correctness of data entered? Another question : "You might also want to capture your users' path" --> yes of course, but giving a set of tasks to users, I don't think that someone can leave the cart while checking out, except if there are some problems which can lead to cart abandonment or stop the process (for example, users don't know how to proceed to payment). Mar 3, 2014 at 14:12
  • By correctness of data, I mean "are they entering the correct data in the proper fields?" Prompts can be confusing: is it asking for name in the blank above or below the label? Does it want their full name, or their given name, or their surname? Also, fields could be in an unexpected order for the target culture. Mar 3, 2014 at 18:23
  • Also be frugal with "giving a set of tasks." If they see the design documents or test instructions, they'll know what's expected from a source other than your site, invalidating your tests. Better to feed them very generic tasks, "You have about $500 to spend, research the 'best' laptop on our site, then buy it on line and ship it to your house. Use credit card XXX." Yes, some tests can be more tightly focused on one part of the experience, but you need to witness the end-to-end navigation. Mar 3, 2014 at 18:29
  • Thanks for your reply. Then , do you mean that I must run two test's session, one before with early users, and then another one to test a larger sample of users, and where take notes about timing against benchmark metrics? I totally agree with you about giving generic tasks. I think that the experience has to be as close as possible to real-world situation. Mar 4, 2014 at 15:46
  • I see running the first set as creating the initial benchmarks. Then, as further changes are made later on (maintenance, adding features, etc), rerun the tests to see if time increased or decreased, and review the changes accordingly. (I base usability on averaging my user timings, and because my app is employee-facing, faster users are better.) When average times increase without an additional feature causing them, it typically means someone made a mistake in maintaining the existing code. Knowing past results helps identify those mistakes. Mar 4, 2014 at 17:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.