Which approach do you usually take when a majority (5 of 6) of users can easily complete a task during usability testing while one user fails? I cannot really describe the task exactly but it would be great to know a general approach that you have used and which was useful. As far as I know the user experiencing the problem wasn't in any particular way different from other users so I cannot just throw out the result as an outlier.
asked Feb 28, 2013 at 0:16
-
" users can easily complete a task during usability testing" I guess it was a summative test, wasn't it?– FrankLFeb 28, 2013 at 12:18
Add a comment
|
5 Answers
Is your question "Should we go ahead and ship the product with a 17% failure rate?" If so, and if you're sure there are no weird extenuating circumstances that compromised the integrity of the test itself, here are the factors I would weigh:
- What does the task failure mean for the user? Is the nature of the task such that a slight inconvenience -- and maybe a guidepost to put him on the right track -- would keep him happy? Or does the failure make him really displeased? (Bills not getting paid on time, uncertainty about whether a purchase was completed, etc.)
- What does the task failure mean for the business goals? Does it prevent the user from giving money to the company? Does it make the user not want to use the product?
For each of these questions, it should be pretty clear whether a 17% failure rate is acceptable or not.
If your question is "How can I get useful information out of his failure?", then I'd focus some energy on determining why he failed. Does he fundamentally misunderstand the way you've structured data? Did he see a cue that wasn't really a cue, and it sent him in the wrong direction?
If you can get that information by interviewing him, that might be a good route to take. If you can't interview him for whatever reason, try testing some more people that you can interview. With a 17% failure rate, it shouldn't take too long to replicate the results. And if you can't replicate them, then your failure rate is actually much lower than 17% -- and therefore a lot smaller problem.
Of course, once you figure out what sent your user astray, you can go about fixing the actual problem.
Most usability tests are not science. They are there to inform your design process and hunt for bugs. If that sixth user had not had a problem your test would have failed, not succeeded (as you seem to think). You are not testing whether your application is user-friendly, the premise of a usability test is that it isn't, and you want to find out why. The five "successes" are where your test went wrong.
As for the consequences for the parts of your design that you did test, they are obviously relatively sound. If the test was conducted properly, you will have some indication of where the sixth user struggled, and what they were thinking.
The trick here is to improve your design without changing the main flow. For instance, when struggling, 80% of users may scan the menus when looking for a way to solve a task, where 20% will try the context menu or the icons. If the task has an entry in the menu bar you've got the 80% covered, but you'll need to find a way to help the 20%. Consider rethinking the context menu, or adding icons to the toolbar. Perhaps you can change the labels in the toolbar slightly to add more scent. It really depends on the specifics of the task.
The point is, you find a way to help the 20% without hurting the 80%.
Q - Usability results: 5 users fly through UI, 1 user fails. What to do?
Ans - Test more people. 6 users isn't really enough unless you are getting the exact same result with all of them.
-
OK but what if you still get the same results - around 17% of people still can't grasp what appears to be simple?– JonW ♦Feb 28, 2013 at 12:03
-
2This is bad advice. Usability bugs block other bugs. If you have a budget for twelve tests, then two sessions of six with a redesign in between are much better than one big session of twelve. If you want to test something quantifiable like conversion rates or efficiency, then you can be more scientific, but for design tests small sessions are better.– PeterFeb 28, 2013 at 17:37
-
@Peter but for me it sounds like quantifiable summatvie test rather than a formative iterative test. Why would you test for task success, if not in a summative test? Even if the answer is short and without arguments, I think it is reasonable.– FrankLFeb 28, 2013 at 18:35
-
JohW - I'd raise it as a major issue if a/ I was getting a 20% failure rate and b/ The 'Why' probing had pinned it down to a specific issue (such as people who enter credit card details with spaces versus those who take the spaces out). The point of running some more users is that you both get a bigger sample - and sometimes it can take the moderator a while to figure out what the right question to ask which reveals why the user is doing what they are doing.– PhillipWFeb 28, 2013 at 22:05
-
@FrankL maybe my response was a little hasty and curt. I guess from the description the test it sounds to me like it conflates formative and quantitative testing. I still think the OP should invest in more formative testing rather than more quantative testing. Once you've found a problem that blocks a user from completing a task you're rarely interested in more precise percentages. Figuring out the why is difficult, and having small iterative tests gives you a better chance of pinning down the problem.– PeterMar 1, 2013 at 12:37
I would have thought normal usability test would capture users that respond/interact differently to the user interface, especially if it is not obviously intuitive. Of course, it is possible to set up a test to get the results that you want, but the objective is really to try and reveal any oversight or unanticipated behaviour that you were not able to account for in your original design.
In my experience, failures in UI testing gives me ideas about how to modify or adjust the design to cater for the potential frustrating for the user, or provide clues about where more guidance can be provided. Depending on the level of detail that you are capturing, it is well worth documenting the breakpoints and use that as input for your next design or test.
answered Feb 28, 2013 at 6:47
The failures often provide more information than the successes!
I don't know your exact task but I know that some things will always resound with a minority of users. This is more research than testing but if you asked 10 users is they write reviews for a purchases (as opposed to 'would they' which gives skewed results) then it's normal for no one to say they do. This is because only about 1 in 100 tend to do that kind of activity (higher for some psychographic groups).
It may be down to specialist knowledge that people either know or not, like the universal share icon (a bloby < sign) with the majority do not understand or some other element.
I would explore more if it was a complete failure. 6 users gives you a rough idea of how things are working but never gives absolute results - 10 is tends to work well for user testing but for user research I've conducted 50 contextual interviews on one project and was still finding out more about my group of users.
