What user experience level to target with usability testing?

Question

I'm working on a project where we have the opportunity to test an upgraded version of a system with 4 users. The new version isn't radically different from the current version, but there are some changed task flows and added functionality users have been asking for. We want to test to find any usability problems with a few critical and frequent tasks we've identified - tasks that all users in this user group perform.

Our user base is broad and consists of users who will be completely new to the system (have not used a previous version), all the way out to highly-skilled users who have a lot of experience on the current system version.

I'm wondering what experience level of users we ought to target for our test: Do we test with 4 users whose current system version experience/expertise level is "just below average", or perhaps one from each quartile of the experience/expertise spectrum?

And a related question: if the ax comes down on the project and we have the opportunity to test with only one user, what experience/expertise level should we target?

Answer 1

It is probably best to spread the participants across as many relevant user types as possible. The most valuable in any group of test participants is always the first one, by a large margin. In one of the most famous articles on this, aptly titled Why You Only Need to Test with 5 Users, Nielsen says:

As soon as you collect data from a single test user, your insights shoot up and you have already learned almost a third of all there is to know about the usability of the design. The difference between zero and even a little bit of data is astounding.

When you test the second user, you will discover that this person does some of the same things as the first user, so there is some overlap in what you learn. People are definitely different, so there will also be something new that the second user does that you did not observe with the first user. So the second user adds some amount of new insight, but not nearly as much as the first user did.

The third user will do many things that you already observed with the first user or with the second user and even some things that you have already seen twice. Plus, of course, the third user will generate a small amount of new data, even if not as much as the first and the second user did.

As you add more and more users, you learn less and less because you will keep seeing the same things again and again.

So if you believe that there are significant differences between your user groups, whatever the grouping criterion, it's best to make sure to test each group instead of investing all your resources in one group.

As to having to choose only one user, I would try to identify the most critical one. For instance, if a novice can decide whether to keep your tool or abandon it, I'd make sure that the novice gets the optimal experience. Sometimes that's not the case, and it's more important to preserve existing power users. Sometimes the one you have to impress is not a user at all but a corporate decision maker who likes shiny things, and his first impression will be the deal breaker. I would choose my single participant based on that.

Answer 2

Pick one. If you're undecided, roll a die. No matter which group of users you select, you will learn something and will have an opportunity to improve your system. Any single usability study isn't going to fundamentally change everything about your product. Do this research, learn something from it, make improvements, and use your results to get more time and resources to do more research in the future -- even if it's to repeat the exact same study with a different 4 users in a different user group.

Given that you only have enough time and resources to study 4 users, I would probably choose the user group to which I have the easiest access. You'll get more responses and a higher chance of doing research with more than one participant. I would not test with single users from multiple user groups. Without other similar users for the sake of comparison, you can't tell the difference between a usability issue with your product and something that is unique to that individual (or perhaps even simply unique to when you asked the question -- after all, everyone has good days and bad days). Further, it makes recruiting participants for your research more complicated. For your first usability study, don't overcomplicate it for yourself. You're not only going to learn about your product, you're also going to learn a lot about the process of conducting research about your product that you can use the next time around.

Answer 3

Test for the one who is about to jump from average to power user, the one that is going to convert and give you profit. Your novice users will eventually become average and power users as well. The power users are like fortune tellers, they know what the product needs in the future.