The Practical Argument
You’re still at the wire-frame stage, so I’ll assume the cost of implementing the changes you propose are pretty negligible –maybe a half day of your time personally. So the choice is:
1. Run 5 or so more users on the existing wireframe to see if you see the same problem.
If yes, then you’re confirming there is a problem but you haven’t confirmed the solution --you didn’t actually test the design changes you recommend. You still should run an additional test on the improvements.
If no, then it looks like the existing wireframe is pretty good as is, but you don’t know if your suggestions would make it better or not. You’re settling for a sufficing design.
2. Run 5 or so user on the revised wireframe to see if it solves the problem
If users generally perform better (as you suspect they will) then you have confirmed that the changes you recommend are helpful, and progress is made.
If users generally perform worse, then you’ve confirmed both that there isn’t a serious problem with the original wireframe and your improvements seem to hurt more than help.
If they perform the same (the least likely outcome), then you can choose the design based on other criteria.
Seems to me you’ll learn more from running users on the revised wireframe with little additional cost, so that’s the way to go.
The Statistical Argument
People poo-poo the sample size of five purely on intuition without actually doing the calculations. So here’s an alternative intuition: Small sample sizes make it very unlikely you’ll get an unrepresentative individual. The very fact that the problematic individual appear in your little sample implies people like them are probably relatively common. I mean, what are the chances that with a sample of just five people you’ll get a 1-in-a-1000 outlier that you can safely ignore?
No, seriously, what are the chances? I wasn’t being rhetorical.
Calculate the binomial probability and you’ll find that chance of one or more 1-in-a-1000 individuals appearing in a random sample of 5 is 0.00499 –about 1 in 200. Do you seriously want to bet the success of your product on your usability test result being a 1-in-200 event? One or more 1-in-a-100 individuals appearing in a random sample of 5 is 0.0490. Hey! That’s statistically significant! I say we reject this hypothesis that the one individual who had problems is some wild outlier.
Small sample sizes don’t mean invalid results. Small sample sizes mean imprecise results. If one user in a sample of five has problems (20% of the sample), then, with 90% confidence, the actual proportion of users in the population with problems is between 1.0% and 65.8%. If 10 users in a sample of 50 have problems (still 20%), then, with 90% confidence, the actual proportion of users in the population with problems is between 11.3% and 31.5% --a much smaller “margin of error.” However, either way, it seems pretty likely a non-trivial minority of your users will have a problem with the product.
Is it worth running dozens more users so you can say with 90% confidence if the problematic portion of users is over single-digit percents, but under the majority of users? Or is your time better spent using these dozens of users to iteratively improve the app?
Okay, that question was rhetorical, because we’re assuming to cost of design iterations are cheap at this wireframe stage, so, of course it’s more productive to use your test users to iterate the design, not get stuck on the current design.
The Qualitative Argument
Usability testing isn’t just about the quantitative results. The whole reason for running such a small sample is so you can be there by their side to observe what actually happening. Review your data for the other four users –notes, video, eye-tracking, whatever you have. Write the story of how each user went through the task. See if there are signs of the kinds problems your recommendations would address. It doesn’t have to be overt frustration and complaints. It could be a single mis-click or an almost-mis-click, or a question the users asks, or long pause. It could be just the raw number of clicks if it’s a lot, even if all were the “right” thing to do. Small subtle issues in the majority of your users can be expected to translate into abject failures for a substantial minority of users. And even if they don’t, why be satisfied with a “successful” but clunky UI that most people can just manage through? If the other four users had any kind of confusion or difficulty –even if they quickly figured it out by themselves –then that alone is justification for improving the design.
Is the one user’s disability plausibly related the problems they were having? If so, is the disability common? Common enough you want to accommodate it? If it’s a 1 in 1000 disability, then maybe you can dismiss this user as an unimportant outlier, however unlikely that is probabilistically. On the other hand, if it’s something like ADHD or dyslexia, then that alone justifies improving the design. You wouldn’t put out a design that relied on perfect color vision, would you? Besides, as a general rule (and as you noted in your case), what’s good for users with disabilities is generally good for “typical” users. Those W3C standards? They benefit everyone.
If the user’s disability is not plausibly related the problems, then absolutely do not dismiss their frustration just because the user is “different.”
You don’t need a usability test result to justify following best practices and usability heuristics. It’s just the right thing to do. A lot of them address concerns that don’t come out in a usability test but nonetheless affect the user experience. Consider yourselves lucky you actually got to see their importance in one user.
The Ego Argument
Doesn’t the team want an awesome UI? I mean, it’s very good right now. Maybe 80% of the way there. But aren’t we the awesome team out to make an awesome product? We have this tool called usability testing that’s great at identifying problems, but doesn’t really give solutions. That’s where you teammates come in. I’ve some ideas that might work, but you guys are awesome. You’re up for challenges. You know that nothing good comes without experimentation. Finding a problem in an experiment is a success, not a failure. It means our team now knows something no one else knows. That’s competitive advantage. That’s the path to awesomeness. Look at the Wright Brothers. Like them, you courageously and imaginatively tackle problems with zeal, even the little ones. Here are the problems. Looking at them, it seems reasonable that they’ll affect a lot of people, if not most, right? What can we do to get to awesome?
