Our team is preparing to perform unmoderated usability testing on our website and are looking for any insight on the number of participants we need to recruit to obtain a significant sample to represent the population we are testing.
Add a comment
|
1 Answer
Jakob Neilsen wrote an article in 2000 called "Why You Only Need to test With 5 Users":
https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
As soon as you collect data from a single test user, your insights shoot up and you have already learned almost a third of all there is to know about the usability of the design. The difference between zero and even a little bit of data is astounding.
When you test the second user, you will discover that this person does some of the same things as the first user, so there is some overlap in what you learn. People are definitely different, so there will also be something new that the second user does that you did not observe with the first user. So the second user adds some amount of new insight, but not nearly as much as the first user did.
The third user will do many things that you already observed with the first user or with the second user and even some things that you have already seen twice. Plus, of course, the third user will generate a small amount of new data, even if not as much as the first and the second user did.
As you add more and more users, you learn less and less because you will keep seeing the same things again and again. There is no real need to keep observing the same thing multiple times, and you will be very motivated to go back to the drawing board and redesign the site to eliminate the usability problems.
After the fifth user, you are wasting your time by observing the same findings repeatedly but not learning much new.
-
1
-
@PhillipW He does not mention moderated or non moderated anywhere in the article as a factor - (although I know his research into this was via moderated testing), so I imagine his point still stands - the more people you test, moderated or not, will uncover fewer and fewer new issues, and you will see lots of people repeating the same issues.– SteveDCommented Jan 10, 2017 at 14:56
-
Jakobs did a follow up article in 2012 on this, which adds more context and re-enforces his earlier article: nngroup.com/articles/how-many-test-users– SteveDCommented Jan 10, 2017 at 14:59
-
1Jacobs point is to do several small rounds of testing / improvement / testing, which is where his magic number 5 comes from. If you only have time to do the one round of testing then the optimum number of users is greater than 5. From his graph its about 10-12. For a single hit testing I'd always over recruit to cover for no shows.– PhillipWCommented Jan 10, 2017 at 16:10