Which approach do you usually take when a majority (5 of 6) of users can easily complete a task during usability testing while one user fails? I cannot really describe the task exactly but it would be great to know a general approach that you have used and which was useful. As far as I know the user experiencing the problem wasn't in any particular way different from other users so I cannot just throw out the result as an outlier.
Is your question "Should we go ahead and ship the product with a 17% failure rate?" If so, and if you're sure there are no weird extenuating circumstances that compromised the integrity of the test itself, here are the factors I would weigh:
- What does the task failure mean for the user? Is the nature of the task such that a slight inconvenience -- and maybe a guidepost to put him on the right track -- would keep him happy? Or does the failure make him really displeased? (Bills not getting paid on time, uncertainty about whether a purchase was completed, etc.)
- What does the task failure mean for the business goals? Does it prevent the user from giving money to the company? Does it make the user not want to use the product?
For each of these questions, it should be pretty clear whether a 17% failure rate is acceptable or not.
If your question is "How can I get useful information out of his failure?", then I'd focus some energy on determining why he failed. Does he fundamentally misunderstand the way you've structured data? Did he see a cue that wasn't really a cue, and it sent him in the wrong direction?
If you can get that information by interviewing him, that might be a good route to take. If you can't interview him for whatever reason, try testing some more people that you can interview. With a 17% failure rate, it shouldn't take too long to replicate the results. And if you can't replicate them, then your failure rate is actually much lower than 17% -- and therefore a lot smaller problem.
Of course, once you figure out what sent your user astray, you can go about fixing the actual problem.
Most usability tests are not science. They are there to inform your design process and hunt for bugs. If that sixth user had not had a problem your test would have failed, not succeeded (as you seem to think). You are not testing whether your application is user-friendly, the premise of a usability test is that it isn't, and you want to find out why. The five "successes" are where your test went wrong.
As for the consequences for the parts of your design that you did test, they are obviously relatively sound. If the test was conducted properly, you will have some indication of where the sixth user struggled, and what they were thinking.
The trick here is to improve your design without changing the main flow. For instance, when struggling, 80% of users may scan the menus when looking for a way to solve a task, where 20% will try the context menu or the icons. If the task has an entry in the menu bar you've got the 80% covered, but you'll need to find a way to help the 20%. Consider rethinking the context menu, or adding icons to the toolbar. Perhaps you can change the labels in the toolbar slightly to add more scent. It really depends on the specifics of the task.
The point is, you find a way to help the 20% without hurting the 80%.
Q - Usability results: 5 users fly through UI, 1 user fails. What to do?
Ans - Test more people. 6 users isn't really enough unless you are getting the exact same result with all of them.
I would have thought normal usability test would capture users that respond/interact differently to the user interface, especially if it is not obviously intuitive. Of course, it is possible to set up a test to get the results that you want, but the objective is really to try and reveal any oversight or unanticipated behaviour that you were not able to account for in your original design.
In my experience, failures in UI testing gives me ideas about how to modify or adjust the design to cater for the potential frustrating for the user, or provide clues about where more guidance can be provided. Depending on the level of detail that you are capturing, it is well worth documenting the breakpoints and use that as input for your next design or test.
The failures often provide more information than the successes!
I don't know your exact task but I know that some things will always resound with a minority of users. This is more research than testing but if you asked 10 users is they write reviews for a purchases (as opposed to 'would they' which gives skewed results) then it's normal for no one to say they do. This is because only about 1 in 100 tend to do that kind of activity (higher for some psychographic groups).
It may be down to specialist knowledge that people either know or not, like the universal share icon (a bloby < sign) with the majority do not understand or some other element.
I would explore more if it was a complete failure. 6 users gives you a rough idea of how things are working but never gives absolute results - 10 is tends to work well for user testing but for user research I've conducted 50 contextual interviews on one project and was still finding out more about my group of users.