SUS Scores and Doubts

Question

My team just started using the SUS to baseline our applications against each other. There are some doubts regarding the relation of the questions to web applications and web sites. At times study participants laugh at the questions or don't know how to answer them appropriately for the context of the study they participated in.

Some questions that raise confusion:

1. I think that I would like to use this system frequently.

For auto insurance, policies last for 6 months, so is frequently a relative term? I am not going to go in and get a quote every day, just when I need it.

5. I found the various functions in this system were well integrated.

How does this relate to a website that has a singular task (i.e. getting an auto insurance quote)?

10. I needed to learn a lot of things before I could get going with this system.

Does this mean they needed to learn a lot about the system, or about the product (auto insurance)?

What have others experiences been with the SUS and how participants and stakeholders perceive the results?

Answer 1

As the original author of SUS, I see this sort of query a lot. I think you need to make a trade-off here, based on the fact that I first developed SUS 25 years ago when I was working on a usability engineering programme for the development of integrated office systems running on VAXes.

1) The terminology may not seem to be particularly relevant to any specific modern technology (websites, mobile phones, what-have-you) and people tend to attribute this simply to the sort of systems it was originally used to assess, and the fact that they differ from the sorts of systems and applications in use today. In some ways that doesn't matter anyway, because the individual items are not supposed to be meaningful in themselves. They are selected from an original, much larger pool of items, on the basis that they were the items that, when presented with extreme examples of usable and unusable systems, were the questions that led to the most extreme responses, both positive and negative. (Thus, in theory, examples of systems with less extreme usability characteristics, should lead to more intermediate responses). The sum total of all of these questions leads to a general measure of perceived usability. So you may disagree with the wording of individual items, but they aren't supposed to have diagnostic value in themselves or to relate to the specific features of a particular system that is intended to be used for a particular purpose. If you want that sort of information, you should write a questionnaire that addresses those specific features. However.....

2) SUS is 25 years old and because it was made freely available has been picked up and used in many, many usability evaluations. (Cheapskates! But I'm glad so many people have found it useful). Consequently there's a wealth of information out there about its use and a body of normative data. There have been some excellent studies done looking at its reliability and collecting normative data - in particular Tullis has an excellent paper on the former aspect and Bangor and Kortum have collected data on the use of the SUS over more than a decade.

So it seems to me you have a choice. You can devise your own questionnaire, which may use terminology relevant to the specific technology you're assessing. It won't have a mass of experience and data from other studies that you can compare it to. If all you're doing is comparing one version of a system or application based on a particular technology with a successor version, that may be all you need. But suppose you want to compare, say, a web-based application with one that's based around a different technology? Then you will start to run into the same sorts of problems that you perceive to be the case with SUS.

I've never claimed SUS to be a perfect tool. (I did say in the title of the published version that it was "quick and dirty"). But I think it's proved its worth over the years and the efforts of people like Thomas Tullis and Phil Kortum (who did all of their work quite independently of me) have provided additional evidence that it's a tool worht using.

rgds

John Brooke

Answer 2

In general, SUS seems to be okay for web sites. Tullis and Stetson compared it with other usability surveys in assessing a corporate intranet, and SUS outperformed the others.

It sounds like a couple certain individual items may not be applicable for your work. To check for this systematically, create a correlation matrix of your item responses and calculate Cronbach’s standardized alpha both with and without the problematic items. If alpha is higher without the item than with it, then it is apparently not measuring the same underlying psychological construct as the other items, and you can be justified in simply dropping it (or re-writing it until testing shows it improves alpha).

But before you do, consider the larger context in which you’ll use the scale. Baselining implies you’ll be making comparisons to later scores. Consider that the problematic items may be perfectly suitable for other sites/apps/versions you want to compare to and capture facets of subjective usability that you want to have. For example, maybe in five years your insurance app will include multiple functions (e.g., help the user identify if their autos that are a high risk for theft). Answers to the “various functions” item may be all over the place now, but that’s still useful information that is worth including so scores are comparable to a future multi-function version.

Personally, the items you singled out may have sounded funny, but if I were I user, I think I’d interpret them in a useful way. I’d interpret “use frequently” as relative to the alternatives (e.g., getting a quote over the phone). I’d interpret “well integrated” as how easily I move through the site without a lot of backtracking or data re-entry (e.g., I’m able to specify my car without having to go through umpteen pages). Not needing to “learn a lot” means I can enter my input and get the quote without a lot of work on my part. Yes, that includes both learning the system and the subject matter: It means I can figure out what color you chose for your links, and you provided a clear explanation on what “deductible” means. No, you won’t know if it was trouble learning about the system or insurance that contributed to a low score, if that happens, but you shouldn’t be looking individual item scores for diagnosis anyway. SUS by design only measures one psychological construct, not multiple psychological dimensions.

Answer 3

Wow, John Brook shows up. Cool! Yes, your tool is still very useful & relevant to UX!

---

As for the survey, consider disambiguating the survey by editing the term "system" so the survey addresses exactly what product/feature/service you care about. Also, integration may not apply within getting a quote, but maybe you can ask about the integration of that feature within the larger system.

As far as perceived results by stakeholders, "perceived usability" is the perfect compliment to a larger battery of tests. For example a live A/B test might compare # of errors thrown on checkout between two different designs. usabilitytesting.com might be something you use for prototypes & you can record # of critical errors as a key performance indicator for features in development/design.

If you're the only UX person on staff and you want to pitch a holistic ux testing strategy, some of these won't be possible without help so maybe you can also tie these to head count, vendors, or involve other departments (marketing/research) in the process.

Answer 4

I'll admit up front that I've never used the SUS. For question #2, though, does it matter? A user is unlikely to distinguish between the two, and if you're using this to baseline across applications, what you're getting is a subjective measure of the learning curve. In this case I don't think you can separate the system from the information it presents.

I'm assuming you're observing the sessions, and it seems to me that for at least half the questions on the list you're going to need the qualitative observational data to understand extreme responses.

Answer 5

Even if your website only has one function it probably has all sorts of content. Marketing material, about your company, price plans, terms of service etc. If the only function is still easily found among all this stuff then it is well integrated. If it is lost in the jungle of other content then it is not. :) I think your users will understand this distinction and respond appropriately.

Mr. Brooke, I have often used the SUS and I always recommend it to my Human Factors class.