0

I worked at company that after user testing sessions got users to complete a post testing questionnaire. The questionnaire tended to ask user for an NPS score, answer SUS (usability) questions and the user to pick descriptive terms from the Microsoft desirability tool kit.

Considering that most user testing session have a total of 5 to 7 users is there any value in this approach? And how might we benchmark prototypes and early design against quantitative numbers, with any confidence before it goes through costly development?

2
  • 1
    Yes. It's free data. Ask broad open ended questions, and give them big text boxes to respond with.
    – PhillipW
    Jan 17 at 19:39
  • Backing up what PhilipW says - don't ask Q's they can just say yes or no to!
    – mgraham
    Sep 13 at 9:12
0

is there any value in this approach?

Short answer: YES*

*conditions apply

The Nielsen Norman Group, which is always a great source for research-based guidelines, states that in most cases 5 users are absolutely sufficient to uncover the majority of your usability issues.

The following graph (taken from an older article of the NN Group) shows that after 5 users, the insight gain decreases rapidly. Since it takes time and money to conduct user studies, the ROI for each additional test is close to none. Usability Problems vs number of test users

It is therefore sensible to keep the sample size relatively small for each test. The number of different tests, however, depends strongly on how large your application is and how diverse your user base is.

  • Keep tests short and concise - don't overwhelm your users by testing hundreds of functions with them.
  • And if you have different target groups with different target groups, you should conduct independent tests for those different parts of your system with representative users.

BUT - since you also asked about quantitative data:

I'm afraid 5-7 users are not enough to get any statistically significant data, simply because the sample size is too small.

The NN Group addresses this concern like so:

the vast majority of your user research should be qualitative — that is, aimed at collecting insights to drive your design, not numbers to impress people in PowerPoint.


I would suggest you keep doing tests, focusing more on qualitative data to uncover usability problems and underlying reasons for those problems. After all, I would argue that this might be even more important than quantitative data. It helps you understand not only where problems might lie but why these problems exist for your users. Understanding your users ultimately helps you in building a usable and therefore well-received system.

I'm aware that I didn't address the benchmarking part of your question yet and I'm sorry that my answer might not be satisfactory in that regard. But if you still want to run some numbers, you can always compare the "number of usability problems" that were uncovered in each iteration of your prototype. To get a more informed picture, you could apply "severity of the problem" as a weighing factor.

When your (small) user tests no longer uncover new problems - or only cosmetic/non-severe ones - you can confidently go into a more costly development stage.

For further insights, I recommend you to read the linked articles and also follow the links there.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.