2

I'm designing an edTech product and conducted a usability test with 5 students using interactive wireframes. One student, who had been identified during recruitment as having a learning disability, became frustrated very early on in the UT due to a multi-step onboarding experience that the student found tedious and repetitive.

I made recommendations based on just that one student's experience, but also knowing these improvements would likely improve the experience for all students (as the recommendation follows best practices and usability heuristics.) The team I work with suggested we do more studies before making changes based on one user's struggle.

What are your thoughts and experiences on basing changes on seeming outliers? Do more testing on the same exact wireframes? Or fix the problem and test again (my usual UX process for iterating.)

Part of the issue is that the team I'm working with is very sensitive to critiques and constructive criticism (or anything that recommends a revision to the work they contributed--which is incredibly valuable work!) So the solution might simply be in better communication between myself and them to explain my reasoning for changing something even if it is based on a single user.

2

The Practical Argument

You’re still at the wire-frame stage, so I’ll assume the cost of implementing the changes you propose are pretty negligible –maybe a half day of your time personally. So the choice is:

1. Run 5 or so more users on the existing wireframe to see if you see the same problem.

  • If yes, then you’re confirming there is a problem but you haven’t confirmed the solution --you didn’t actually test the design changes you recommend. You still should run an additional test on the improvements.

  • If no, then it looks like the existing wireframe is pretty good as is, but you don’t know if your suggestions would make it better or not. You’re settling for a sufficing design.

2. Run 5 or so user on the revised wireframe to see if it solves the problem

  • If users generally perform better (as you suspect they will) then you have confirmed that the changes you recommend are helpful, and progress is made.

  • If users generally perform worse, then you’ve confirmed both that there isn’t a serious problem with the original wireframe and your improvements seem to hurt more than help.

  • If they perform the same (the least likely outcome), then you can choose the design based on other criteria.

Seems to me you’ll learn more from running users on the revised wireframe with little additional cost, so that’s the way to go.

The Statistical Argument

People poo-poo the sample size of five purely on intuition without actually doing the calculations. So here’s an alternative intuition: Small sample sizes make it very unlikely you’ll get an unrepresentative individual. The very fact that the problematic individual appear in your little sample implies people like them are probably relatively common. I mean, what are the chances that with a sample of just five people you’ll get a 1-in-a-1000 outlier that you can safely ignore?

No, seriously, what are the chances? I wasn’t being rhetorical.

Calculate the binomial probability and you’ll find that chance of one or more 1-in-a-1000 individuals appearing in a random sample of 5 is 0.00499 –about 1 in 200. Do you seriously want to bet the success of your product on your usability test result being a 1-in-200 event? One or more 1-in-a-100 individuals appearing in a random sample of 5 is 0.0490. Hey! That’s statistically significant! I say we reject this hypothesis that the one individual who had problems is some wild outlier.

Small sample sizes don’t mean invalid results. Small sample sizes mean imprecise results. If one user in a sample of five has problems (20% of the sample), then, with 90% confidence, the actual proportion of users in the population with problems is between 1.0% and 65.8%. If 10 users in a sample of 50 have problems (still 20%), then, with 90% confidence, the actual proportion of users in the population with problems is between 11.3% and 31.5% --a much smaller “margin of error.” However, either way, it seems pretty likely a non-trivial minority of your users will have a problem with the product.

Is it worth running dozens more users so you can say with 90% confidence if the problematic portion of users is over single-digit percents, but under the majority of users? Or is your time better spent using these dozens of users to iteratively improve the app?

Okay, that question was rhetorical, because we’re assuming to cost of design iterations are cheap at this wireframe stage, so, of course it’s more productive to use your test users to iterate the design, not get stuck on the current design.

The Qualitative Argument

Usability testing isn’t just about the quantitative results. The whole reason for running such a small sample is so you can be there by their side to observe what actually happening. Review your data for the other four users –notes, video, eye-tracking, whatever you have. Write the story of how each user went through the task. See if there are signs of the kinds problems your recommendations would address. It doesn’t have to be overt frustration and complaints. It could be a single mis-click or an almost-mis-click, or a question the users asks, or long pause. It could be just the raw number of clicks if it’s a lot, even if all were the “right” thing to do. Small subtle issues in the majority of your users can be expected to translate into abject failures for a substantial minority of users. And even if they don’t, why be satisfied with a “successful” but clunky UI that most people can just manage through? If the other four users had any kind of confusion or difficulty –even if they quickly figured it out by themselves –then that alone is justification for improving the design.

Is the one user’s disability plausibly related the problems they were having? If so, is the disability common? Common enough you want to accommodate it? If it’s a 1 in 1000 disability, then maybe you can dismiss this user as an unimportant outlier, however unlikely that is probabilistically. On the other hand, if it’s something like ADHD or dyslexia, then that alone justifies improving the design. You wouldn’t put out a design that relied on perfect color vision, would you? Besides, as a general rule (and as you noted in your case), what’s good for users with disabilities is generally good for “typical” users. Those W3C standards? They benefit everyone.

If the user’s disability is not plausibly related the problems, then absolutely do not dismiss their frustration just because the user is “different.”

You don’t need a usability test result to justify following best practices and usability heuristics. It’s just the right thing to do. A lot of them address concerns that don’t come out in a usability test but nonetheless affect the user experience. Consider yourselves lucky you actually got to see their importance in one user.

The Ego Argument

Doesn’t the team want an awesome UI? I mean, it’s very good right now. Maybe 80% of the way there. But aren’t we the awesome team out to make an awesome product? We have this tool called usability testing that’s great at identifying problems, but doesn’t really give solutions. That’s where you teammates come in. I’ve some ideas that might work, but you guys are awesome. You’re up for challenges. You know that nothing good comes without experimentation. Finding a problem in an experiment is a success, not a failure. It means our team now knows something no one else knows. That’s competitive advantage. That’s the path to awesomeness. Look at the Wright Brothers. Like them, you courageously and imaginatively tackle problems with zeal, even the little ones. Here are the problems. Looking at them, it seems reasonable that they’ll affect a lot of people, if not most, right? What can we do to get to awesome?

2
  • This answer is brilliant and answers every single aspect of my problem. The way you broke it down will hopefully help me improve my communication with the team based on how each of them think. Thank you so much!
    – MRL
    Jul 16 at 5:38
  • I don't agree with your statistics argument: if you get an unlikely event either it isn't an unlikely event OR randomly you've found the unlikely event. Only by collecting a larger sample size will you answer this question ( think about it in terms of the one red counter in a large bag of green counters)
    – PhillipW
    Jul 16 at 8:37
0

My view is that 5 users isn't enough to draw robust conclusions. And I know Mr Neilsen disagrees.

https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

I'd push the sample up to 10-15. The more variation (gender, age, experience) in the users you recruit, the bigger the sample required.

"Sample of 1" data doesn't prove very much.

3
  • So the way I'm looking at that is that if it's just one person having an issue, it could potentially be ignored? My feeling is that even if we test 100 people and that 1 person is having issues, that a change could be warranted depending on the severity. Is that incorrect? (Note: this issue especially is relating to education and disability. So if a disabled person is the only one having the issue, could failing to address that be an issue of inequity or even legality?
    – MRL
    Jul 15 at 19:40
  • I wouldn't ignore it. It's still data. However I'd test more people (say 5 more). If the pattern doesn't show up again then you can say that a subset of users have this issue and you have a better idea how often it crops up. Which effects how you 'load' the results. If you are still finding new issues by user 10, then you still haven't hit the 'point of repetition'. Whether you implement changes based on all findings is a business decision.
    – PhillipW
    Jul 16 at 7:25
  • BTW The point of repetition is called 'Saturation in market research intotheminds.com/blog/en/…
    – PhillipW
    Jul 16 at 7:33
0

It sounds like you've uncovered a cognitive accessibility issue. Generally in American education, if even one student with a disability can't complete a task because of inaccessibility, it can create a liability for the institution.

It sounds like your tester was able to complete the (hopefully one-time) task, even though it was a sub-optimal experience. The onboarding experience might not cause litigation, but it might harm your company's reputation or affect sales. Accessibility is a broad swath of guidelines that assist people from many different backgrounds; even having a carousel on your homepage that can't be paused is an accessibility issue.

Your team's sensitivity to any negative feedback is an issue, to be sure. It sounds like they need to develop far more empathy. Having them meet the people who are struggling with their products, and watching them use the products, might be helpful.

1
  • 2
    Thank you so much for your response and the links. I'm using them along with the accepted answer to try to help the team understand the severity levels of usability issues. You've been a great help!
    – MRL
    Jul 16 at 5:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.