To collect metrics, I recommend using a very simple
usability measure: the user success rate. I define this rate as the
percentage of tasks that users complete correctly. This is an
admittedly coarse metric; it says nothing about why users fail or how
well they perform the tasks they did complete.
Nonetheless, I like success rates because they are easy to collect and
a very telling statistic. After all, if users can't accomplish their
target task, all else is irrelevant. User success is the bottom line
Success rates are easy to measure, with one major exception: How do we
account for cases of partial success? If users can accomplish part of
a task, but fail other parts, how should we score them?
Let's say, for example, that the users' task is to order twelve yellow
roses to be delivered to their mothers on their birthday. True task
success would mean just that: Mom receives a dozen roses on her
birthday. If a test user leaves the site in a state where this will
occur, we can certainly score the task as a success. If the user fails
to place any order, we can just as easily determine the task a
But there are other possibilities as well. For example, a user might:
order twelve yellow tulips, twenty-four yellow roses, or some other
deviant bouquet; fail to specify a shipping address, and thus have the
flowers delivered to their own billing address; specify the correct
address, but the wrong date; or do everything perfectly except forget
to specify a gift message to enclose with the shipment, so that Mom
gets the flowers but has no idea who they are from. Each of these
cases constitutes some degree of failure (though if in the first
instance the user openly states a desire to send, say, tulips rather
than roses, you could count this as a success).
If a user does not perform a task as specified, you could be strict
and score it as a failure. It's certainly a simple model: Users either
do everything correctly or they fail. No middle ground. Success is
success, without qualification.
However, I often grant partial credit for a partially successful task.
To me, it seems unreasonable to give the same score (zero) to both
users who did nothing and those who successfully completed much of the
task. How to score partial success depends on the magnitude of user
In the flower example, we might give 80% credit for placing a correct
order, but omitting the gift message; 50% credit for (unintentionally)
ordering the wrong flowers or having them delivered on the wrong date;
and only 25% credit for having the wrong delivery address. Of course,
the precise numbers would depend on a domain analysis.
There is no firm rule for assigning credit for partial success.
Partial scores are only estimates, but they still provide a more
realistic impression of design quality than an absolute approach to
success and failure.