0

We are going to conduct a usability testing with a small sample (5 participants) as part of an iterative design process meaning we understand that the whole idea of 5 participants is to gather feedback, improve the design and perform more usability testings in future iterations.

But my question is: what should be reported to the development team when you have a small sample of 5 participants?

More in detail what I mean with my question:

  • Should we just report usability problems observed (qualitative data such us: "3 participants clicked in the wrong button when trying to sign in") or can we report task completion, time on task, SUS, etc. (for example: "task completion was 60% for task 1"?
  • Or are these metrics (task completion, time on task, etc) only for quantitative usability testing where you have a bigger sample of 30+ participants?

Thank you in advance,

1
  • You are mixing up qualitative and quantitative results. With 5 participants it is hardly useful to mention the numbers. You can report results something in the line of "We observed that ... will be a problem for some users". As it is just a sample you can't really say that it "is" a problem but that can say with certainty that it (probably) "will" be a problem. It depends on the kind of problem how big the impact will be, no numbers needed for that, just professional judgement.
    – jazZRo
    Commented May 12, 2022 at 7:43

4 Answers 4

2

Five participants can uncover usability issues, but you can't report confidently with a sample size that small, so stay away from quantifying it. You can give the development team your suggestions for improvement (no need for 30 people to discover that the same button is broken), but if you'll be measuring improvement on the same feature at a later date, you really need a statistically significant sample for that (30 at the very least, 40 is better).

A participant pool size of 5-12 is good for qualitative feedback. You might use 5 participants to find out why your analytics data are reporting that something is off, and provide verbatims from the users that support what you discover.

2

First of all you should definitely check Nielsen Norman's article named "Why You Only Need to Test with 5 Users" (link)

Basically you can use it to find usability problems or validate your assumptions but you shouldn't make assumptions for overall trust or time on site and stuff. That's why normally there is a 2 phase of testing a design; first phase is qualitative which would help you to deliver usable design for most of the users and second quantitative which would help you to find out overall numbers that you asked. We do a/b tests to get those numbers on the second phase.

So I'd say you can track task completion as well (there might be a usability problem behind it) along with other usability issues.

1

It can depend on context, but generally I find that small sample size testing like that is most useful while the development is in early to mid stages. So the most valuable feedback tends to be that which will validate or invalidate assumptions about which tasks are most important to do sooner rather than later. On all but the smallest projects, there will always be many things to do as well as uncertainty about what should be done first, where the most ideal trade-offs lie, etc. The larger the gulf in domain expertise between developers and users, the more assumption validation and learning there needs to be. More metric-oriented reporting tends to be less critical at this stage where the product is churning rapidly, and is not terribly reliable for small sample sizes anyway.

I would try to specifically highlight things that answer the question: "What part of the product needs the most immediate attention to unlock value for the customer and/or reveal how best to expend dev resources for the next work cycle."

Be aware that that this can sometimes run into political issues, particularly when testers are expected to just report findings and others are left to make strategic determinations about resource allocation. But at the end of the day that is where the essential value of early cycle rapid testing lies. It can be beneficial (and humbling!) to have us developers ourselves silently observe these small early test sessions periodically, and not just rely on reports of test observed by others.

Typically, at the end I also like to ask what the participant felt were the "highs" and "lows" of their experience. Their answers will sometimes not match what I would have assumed based on my observations of their session, but those cases almost always provide key insights into the how a user's perspective, expectations, and mental model differs from those actively involved with the project development.

0

The report should include what you were trying to measure and the results of those tests. Your bullet point could be the quantitative analysis such as "60% of users clicked X button" when trying to do "y" and then you can include the actual observations and quantitative data like direct quotes. The data from the tests and the qualitative observations helps the team understand what the problems are and what you observed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.