Giving customers a “usability score”?

Question

Giving a (software) product a numeric usability score is a thought I had in many of my past projects. Having a final verdict in form of a %-value or a grade would give me a certain comparability between projects and some customers would prefer this kind of "certificate" over (just) an elaborate text. While for full reviews I still prefer the qualitative, informal approach, this formal summary could be applied when a full review isn't demanded.

Userfocus.co.uk has a nice example for this; guidelines can be checked (3-tier: doesn't comply, complies and kind of complies) and the XLS will promptly output a %-score for all of the 9 topics, together with a beautiful spiderweb-chart.

I gave it a try on my companys website. My impressions were mixed, it did provide some tendencies where the main problems are, but the statistical value was very dubious in my eyes. Some of the topics had only few checkpoints relevant to the website and some of the questions were highly subjective to anser. E.g. the topic "Forms & Data Entry" has a full 100% but mainly because only 6 of 23 answers were given.

What do you think? Does this quantitative approach has any value for evaluation? Could it be done right to get a meaningful and statistically valid outcome?

Answer 1

A single evaluation cannot be statistically significant. If you need statistical significance, then you shouldn't be doing a heuristic evaluation or expert review.

That doesn't mean that giving a numeric answer doesn't have value for a heuristic evaluation or expert review. It has value as a communication tool, which is what you say your customers are asking for. Numbers also have risks as communication tools. We assume that numbers are absolute, so something that has a score of 7 on a 10-point scale is significantly better than something that has a score of 5 on that same scale. Further, any scale that is close to a scale that is used in school grading also has risks because of the implicit meaning in those numbers (for example, a 4.0 scale or a 100-point scale has implicit meaning to an American audience).

I do not think that a numeric usability score is formal and a textual report is informal. The goal of the report, regardless of its format, is to communicate with the stakeholders about what works and what doesn't in a given design. A number is not a formal way of communicating this information to your stakeholders, and an assumption on anyone's part that a number is more formal than text is another risk of reporting a number. Further, reporting a number does not mean that you're doing quantitative work. Instead, you're assigning a qualitative meaning to a number (or range of numbers), which can introduce confusion about whether the work that you have done is actually qualitative or quantitative in nature.

If statistical significance is a requirement, then I recommend reading "Measuring the User Experience" by Thomas Tullis and William Albert. This means that you're no longer doing heuristic evaluations, but there are some relatively quick and low-cost methods to gather statistically-significant data.

Answer 2

I personally don't find expert reviews, or heuristic reviews, that useful as general tools. There are too many exceptions and context dependent issues.

For example in the survey you point to one of the questions is "Navigation-only pages (such as the home page) can be viewed without scrolling" that immediately has me asking:

• Scrolling on what (phone? tablet? netbook? laptop?)
• What about long reference pages that my experience with user testing tells me people are likely to prefer.
• What is the solution? Splitting a badly designed IA so that long pages of links are pushed to many small pages of links will likely make things worse.

There are similar sort of problems with many of the questions.

If you want a number - look to things like the System Usability Scale or the Net Promoter Score that are based on what users think of the system, rather than heuristics that we hope correlate well with what the users think of the system.

Answer 3

I would say this is a good idea. Although in the past a five start method has worked just as well for me (with maybe half stars). It's also common to use a traffic light system to show what is good, okay but needs tweaking and bad.

In past expert reviews giving different aspects a score generates a set of results that can be viewed at a glance. For this to make more sense comparing the experience of the given product with others in the market shows who well/badly the product being reviewed is.

The criteria for an expert review should fit the problem. In the past I have broken reviews down into functionality areas and then overlapped this with overall issues like focus , consistency, content, language etc.