How to measure Success rate?

Question

I'm reading this book: Tullis, Thomas, and William Albert. Measuring the user experience: collecting, analyzing, and presenting usability metrics. Morgan Kaufmann, 2010.

In this book, 5 usability metrics are defined, one of them is success rate. I want to know how to measure the success rate? is that equivalent to a usability attribute or a usability metric is something different?!

Answer 1

A success rate is one of many metrics used for measuring/quantifying usability. As http://www.measuringusability.com/blog/essential-metrics.php describes, if a task can not be completed, the product is not usable. "If users cannot accomplish their goals, not much else matters."

The methods used for determine success rate will vary greatly depending upon:

1. What is the task being performed?
2. How is the task completed?

Some task examples and how to measure their success rates:

• Search

  • A user goes to searchengine.com
  • They search for 'pizza'
  • The user clicks the first web result

  Success rate: This would be considered a success because the user found and navigated to a web result.

  • The user searches for "2+2"
  • The answer '4' is given right away
  • The user does not click a result, but their task was successfully completed

  Success rate: Detecting a success like this is more difficult, but might be detected by things such as knowing that a similar query was not performed directly after (this would indicate that the answer was not found so the user asks the same thing in a different way in attempt to find the answer).

• Draw

  • The user goes to a drawing website
  • The user clicks 'draw' and is presented with a canvas
  • Content is added to the canvas and the drawing is submitted

  Success rate: If the user starts the drawing task, but does not submit, this would not be successful. A successful submission of content would add to the success rate in this case.

Points to keep in mind:

• Depending on the task, the success or completion might be determined in multiple ways
• First determine exactly what is the task
• Next, how do we know the task was completed
• It may be useful to track other actions as well, to determine why a task was not successful.
  • For example, in the search example, if you were to track time spent on the page, then you would know if there was an immediate re-query, or if the user read the content previews first.
    • An immediate re-query might suggest that the page was intimidating, unorganized, or unattractive to read,
    • while time spent on the page might suggest that the content presented was not relevant to the query.

Answer 2

You need to set your business goals and based on that you can set you ways to measure the success.

Think about an e-commerce site, or think about a social network. The goals of the business will be completely different, i.e. % abandoned baskets, conversion, etc.

Tools like Google analytics help to make sure you are able to constantly monitor the site and improve the success as the product becomes more mature.

When you create your usability testing criteria, make sure you consider what is it that you want to accomplish.

Answer 3

You can have several levels of "success rate", but in essence it's a matter of saying "yes" or "no" to this question:

"Did the user accomplich the task?"

---

Quote from the book, p65:

To measure task success, each task [...] must have a clear end-state.

---

It's similar to "effectiveness" in the ISO 9241-11 definition...

Answer 4

Nielsen Norman Group:

Measuring Success

To collect metrics, I recommend using a very simple usability measure: the user success rate. I define this rate as the percentage of tasks that users complete correctly. This is an admittedly coarse metric; it says nothing about why users fail or how well they perform the tasks they did complete.

Nonetheless, I like success rates because they are easy to collect and a very telling statistic. After all, if users can't accomplish their target task, all else is irrelevant. User success is the bottom line of usability.

Success rates are easy to measure, with one major exception: How do we account for cases of partial success? If users can accomplish part of a task, but fail other parts, how should we score them?

Let's say, for example, that the users' task is to order twelve yellow roses to be delivered to their mothers on their birthday. True task success would mean just that: Mom receives a dozen roses on her birthday. If a test user leaves the site in a state where this will occur, we can certainly score the task as a success. If the user fails to place any order, we can just as easily determine the task a failure.

But there are other possibilities as well. For example, a user might:

order twelve yellow tulips, twenty-four yellow roses, or some other deviant bouquet; fail to specify a shipping address, and thus have the flowers delivered to their own billing address; specify the correct address, but the wrong date; or do everything perfectly except forget to specify a gift message to enclose with the shipment, so that Mom gets the flowers but has no idea who they are from. Each of these cases constitutes some degree of failure (though if in the first instance the user openly states a desire to send, say, tulips rather than roses, you could count this as a success).

If a user does not perform a task as specified, you could be strict and score it as a failure. It's certainly a simple model: Users either do everything correctly or they fail. No middle ground. Success is success, without qualification.

However, I often grant partial credit for a partially successful task. To me, it seems unreasonable to give the same score (zero) to both users who did nothing and those who successfully completed much of the task. How to score partial success depends on the magnitude of user error.

In the flower example, we might give 80% credit for placing a correct order, but omitting the gift message; 50% credit for (unintentionally) ordering the wrong flowers or having them delivered on the wrong date; and only 25% credit for having the wrong delivery address. Of course, the precise numbers would depend on a domain analysis.

There is no firm rule for assigning credit for partial success. Partial scores are only estimates, but they still provide a more realistic impression of design quality than an absolute approach to success and failure.