User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It only takes a minute to sign up.
Sign up to join this community
On SUS - the question "I think that I would like to use this system frequently" does not make sense for sites which are used infrequently, for example a mortgage payment site.
How does this question reflect the overall usability of the website? Ideas?
The System Usability Scale is mainly intended for a comparison between like systems. It's very general to allow it to compare systems, and it gives an easy reference point. It's generally assumed that the products you are comparing have similar use cases, which mitigates the problem of some questions not applying.
Comparing mortgage payment sites the question "I think that I would like to use this system frequently" is likely to be rated negatively for both similar systems. The scores presumably still be comparable for both systems because the question is equally poor for both products.
As with all cut and dried evaluations, the SUS is only a general tool and you must consider each question's validity to your use-case. You can't use it as the only analysis of usability; it's a great tool for comparison, but much too general for any single item.
If you're conducting your own usability analysis and not comparing two products, feel free to omit some questions if you feel they are confusing or irrelevant when applied to your product. Create a heuristic review that applies to your product. Here's a great Heuristic Evaluation Checklist with a great deal of questions. It's not a likert scale nor is it worded for an end user, but you can get a good idea of some questions you can ask to evaluate your system.
SUMI is a more comprehensive usability scale, practically the opposite of the "quick and dirty" SUS. SUMI can give you a lot more data but again, consider constructing a brief, personalized questionnaire for your product if you feel no questionnaires apply. SUMI is quite deep and a briefer survey is often easier to get people to complete (and complete honestly).
In this type of questionnaires, the statements (called “items” in the technical literature) are not chosen mainly because they describe what usability is but because they are affected by it (i.e. the question is not “how does this affect usability” but “how does usability affect this”) and the ratings are correlated together. For the SUS, they were originally selected from a pool of 50 candidate statements based on a small study with two systems that could have been used frequently by the participants, which might explain why this question was retained.
If a statement does not seem to relate to the usability of the website for the participants, you would expect little variability, no differences between products regarding this particular statement and no correlation with the rest of the questionnaire. Technically speaking, it could limit the theoretical maximum score you could obtain with the questionnaire (i.e. bias) or contribute some random error to the total score. This would in turn slightly limit the power to detect differences but, as Ben noted, it would not bias a comparison between two websites in the same category (previous iteration of your design, competitors…)
It also conceivable that some respondents see through it and answer anyway based on their perception of the website itself, not the task it is designed to support. However, if you have several such irrelevant or awkwardly formulated statements in a questionnaire, the respondents might react negatively, complain about it or become less cooperative in general (I have seen it happen with the SUMI).
If you want to know more about the SUS, the original reference is
I also gave some other interesting references in this answer to another question about the SUS You can also check this question which raised similar issues and even includes an answer from the creator of the SUS.
