What is the acceptable success score for user testing?

Question

I'm just wondering how high of a success rate does the outcome of a user-testing have to be before you can consider it successful. I know that 100% is the target but if time and budget are limited, what's the minimum acceptable score? Given that 100 is the highest possible score.

Answer 1

Why do you want to test?

Typically, usability analysts test a product's usability to see if the mental model works, if the workflow works, if the interaction (controls) are discoverable, if a task can be completed within a certain time, how many participants succeed at a given task (which is what you're after, I think), and so on.

The answer, as usual, is "It depends."

• How critical is the task? (What's at stake if the user fails?)
• How fast must the task be completed correctly to be deemed a success?
• If the user fails, will they recognize the failure or will they believe the task succeeded?
• If the user can recover from a failure (an error state), is that success with a longer time on task, or a failure?
• And so on.

It's not unreasonable to expect a very high success rate. But it's also not uncommon to get lower success rates, if the GUI or design needs to be fixed.

In case you're new to usability testing, Gabriel, I'll add one tip. Make sure you give your participants realistic scenarios that don't lead the participants toward the desired outcome. For example, don't ask the users to "click the Save button," but instead ask them to "make sure you can use this document again, when you next use this program."

Answer 2

It is a teeny weeny bit analogous to service level agreements (SLAs), or disaster recovery...."how much do I spend"....and the answer, with regards to UX matters, is "what's the cost to your organisation of failure".

If your application is mission critical (e.g. nuclear power station control panel) or a medical device (e.g. X-ray machine) then the cost of failure is massive and the pass levels need to be nigh on 100% for all users to prevent lethal mistakes.

If your application is eCommerce related, I'd put the success rate up at 90% or somewhere reasonable around there. You need your users to be able to achieve organisational goals (e.g. buy stuff, meet targets/plans etc) but also need to recognise that the audience will be significantly different from a medical or military application.

Much less than 90% and you have to question whether your interaction design is appropriate....imho :)

N

Answer 3

How are you even scoring "success"? The purpose of user testing is to reveal weak points in a design, not to get a high score. Reducing the results to a number makes the entire process useless; why not just put it up on Mechanical Turk and see what comes of it?

In fact, you could argue if you get a good 'score', you are wasting everyone's time. It means the testing was done much too late in the process to be useful, or was unnecessary at that stage.

IMHO the most useful usability tests are the ones where users do get through all the tasks, but reveal many previously unknown things about your system along the way.