How many users should be tested with

Question

Traditionally I've always stuck to the number 10. I recall reading some pretty convincing research that said 10 is the point at which 90% of problems will have been discovered and the gain from adding more users begins to level off.

Today I go to search for an article on this so I can include it as a reference as to why we used 10 people in our test though...and I find Nielsen is now saying 5 is the number. And apparently has been for a decade. Total case of mis-remembering on my part?

But I search further...and find an article saying 20 is the perfect number if you can manage it!

And then here's one that says 15!

Apparently everything but the 10 that has been my standard for years and has got me through life thus far...I guess I always spoke convincingly enough that people just took what I said as a given.

So many conflicting numbers. Which is right?

Answer 1

For me there is no "perfect" number. It's more about the budget I have for testing, the timescale I have for testing, and how the results of the testing is going to be applied, and so on.

For example:

• All things being equal I'd prefer to do multiple small-tests rather than one large test. So we can demonstrate we have fixed the things we discover in test A during test B, the ones we discover in test B in test C, and so on. It also allows us to find the new problems that we introduce by fixing the problems we found earlier.
• There is no point in finding 99% of the problems, if the team only has enough time and resources to fix 20% of them. So doing a large usability test is going to be mostly waste, and we can better spend those resources elsewhere.
• If we're in a situation where we only have access to a user group for a limited period of time then maybe it's worth spending more time and resources on a larger test so we have more things to do during the fallow period where we don't have access to users.
• We might want to do smaller tests at the start of product development — when we're mostly finding boulder sized problems — and larger tests at the end — where we're mostly finding pebble sized problems that smaller groups might miss.
• and so on…

The right number is completely context dependent.

Since I spend most of my time with smaller teams doing more iterative agile/lean style development I tend to do lots of smaller tests. 3-5 people every other week. The largest test I ever did was about 40 people (5ish a day over two weeks) where we had a one-shot opportunity to do some comparative usability tests. Both numbers were "right".

Answer 2

The Nielson Norman Group has several articles on the subject.

In the article "Why You Only Need to Test with 5 Users" the following is discussed:

Summary: Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests as you can afford.

Where they show the diminishing returns on testing more and more people.

However, in "How Many Test Users in a Usability Study?" they point out that there are always exceptions.

As with any human factors issue, however, there are exceptions:

Quantitative studies (aiming at statistics, not insights): Test at least 20 users to get statistically significant numbers; tight confidence intervals require even more users.

Card sorting: Test at least 15 users.

Eyetracking: Test 39 users if you want stable heatmaps.

However, these exceptions shouldn't worry you much: the vast majority of your user research should be qualitative — that is, aimed at collecting insights to drive your design, not numbers to impress people in PowerPoint.

Answer 3

my thoughts are keep testing until you find consistency in your findings. For major problems this does not take many users (2-5) but the thing that is good about getting consistent results is that it gives you more negotiating power when you come to justify your recommendations.