3

From my research, the SUS seems to be the industry standard when it comes to measuring usability through a questionnaire. There are of course plenty of other instruments.

Nielsen's attributes of usability are widely recognised. From what I have seen, questionnaires and studies rarely use Nielsen's attributes / heuristics directly.

So why not just create a questionnaire with a likert scale and ask the user either the five attributes or the ten heuristics directly? Why do people prefer to use the SUS instead?

3
  • 1
    One thing to consider is that heuristics are typically applied by someone with experience, whereas the SUS can be asked of anyone. In other words, they are different tools for different needs.
    – DA01
    Aug 22, 2015 at 18:12
  • I don't quite follow. What experience does it take to ask five questions that correspond to the five attributes? You could just ask them to rate the memorability of a UI, etc. The only thing that's missing is a conclusive way of interpreting the results. I guess that's why the SUS is so popular? Why has nobody created a measurement instrument that's based on Nielsen's factors? You would think that it should be possible to create a very reliable instrument with high validity. If not, then Nielsen's factors would be wrong...
    – reggie
    Aug 22, 2015 at 19:16
  • When you say industry standard, I have actually never seen any companies that I worked for implement the SUS. So it might be a standard but perhaps not for all industries. As a starting point, the SUS is good to get you thinking about what usability means to your organisation though...
    – Michael Lai
    Aug 23, 2015 at 5:32

2 Answers 2

2

Because SUS works very well and has been examined closely practitioners for more than 25 years.

The SUS questions weren't pulled out of a hat. They were research based. To somewhat extensively quote from Brooke's original paper:

SUS is a Likert scale. It is often assumed that a Likert scale is simply one based on forced-choice questions, where a statement is made and the respondent then indicates the degree of agreement or disagreement with the statement on a 5 (or 7) point scale. However, the construction of a Likert scale is somewhat more subtle than this. Whilst Likert scales are presented in this form, the statements with which the respondent indicates agreement and disagreement have to be selected carefully.

The technique used for selecting items for a Likert scale is to identify examples of things which lead to extreme expressions of the attitude being captured. For instance, if one was interested in attitudes to crimes and misdemeanours, one might use serial murder and parking offences as examples of the extreme ends of the spectrum. When these examples have been selected, then a sample of respondents is asked to give ratings to these examples across a wide pool of potential questionnaire items. For instance, respondents might be asked to respond to statements such as “hanging’s too good for them”, or “I can imagine myself doing something like this”.

Given a large pool of such statements, there will generally be some where there is a lot of agreement between respondents. In addition, some of these will be ones where the statements provoke extreme statements of agreement or disagreement among all respondents. It is these latter statements which one tries to identify for inclusion in a Likert scale, since, we would hope that, if we have selected suitable examples, there would be general agreement of extreme attitudes to them. Items where there is ambiguity are not good discriminators of attitudes. For instance, while one hopes that there would be a general, extreme disagreement that “hanging’s too good” for those who perpetrate parking offences, there may well be less agreement about applying this statement to serial killers, since opinions differ widely about the ethics and efficacy of capital punishment.

SUS was constructed using this technique. A pool of 50 potential questionnaire items was assembled. Two examples of software systems were then selected (one a linguistic tool aimed at end users, the other a tool for systems programmers) on the basis of general agreement that one was “really easy to use” and one was almost impossible to use, even for highly technically skilled users. 20 people from the office systems engineering group, with occupations ranging from secretary through to systems programmer then rated both systems against all 50 potential questionnaire items on a 5 point scale ranging from “strongly agree” to “strongly disagree”.

The items leading to the most extreme responses from the original pool were then selected. There were very close intercorrelations between all of the selected items (± 0.7 to ± 0.9). In addition, items were selected so that the common response to half of them was strong agreement, and to the other half, strong disagreement. This was done in order to prevent response biases caused by respondents not having to think about each statement; by alternating positive and negative items, the respondent has to read each statement and make an effort to think whether they agree or disagree with it.

Brooke's 2013 retrospective paper on the SUS is also worth a read.

I strongly suspect just asking folk questions about the five attributes or the ten heuristics directly won't give useful results. Because, humans. It's darn tricky to find questions that don't bias folk in certain directions or that are interpreted by different people in different ways.

If you do want some more insight into specific factors you might be interested in some of the more recent work that James Lewis & Jeff Sauro have done on a factor analysis of the SUS might be of interest.

0

I guess people like the SUS because it is something well-known and off-the-shelf, in the sense that you can pretty much use it as is or make slight modifications and adapt it to your purpose. SUS has the characteristic that it is generic and can be applied to a lot of different types of products and services, so it is good for comparisons across different industries or even within the same industry.

However, I think that Nielsen's Attributes of Usability, like most attempts to try and gauge usability from a systematic and qualitative analysis, is not so easily adaptable and requires a little bit more thinking in terms of applying them in a questionnaire as well as analysing the results.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.