Can SUS scores be used for comparison between different systems?

Question

The System Usability Scale (SUS) resulted score seem to me as somewhat subjective. What ever score yields from a questionnaire for System A may just solely be just for System A, and is difficult to compare the score with System B's because of the subjectivity.

I searched around and saw this on Wikipedia:

Effectiveness in using a system for controlling a continuous industrial process would generally be measured in very different terms to, say, effectiveness in using a text editor. Thus, it can be difficult, if not impossible, to answer the question “is system A more usable than system B”, because the measures of effectiveness and efficiency may be very different.

So, it seems that this part of the article agrees that it is difficult to make a comparison between systems. However, at the later part of the same Wikipedia article, it says:

SUS has generally been seen as providing this type of high-level subjective view of usability and is thus often used in carrying out comparisons of usability between systems. Because it yields a single score on a scale of 0–100, it can be used to compare even systems that are outwardly dissimilar.

Now, it seems that comparison between systems with SUS score is possible? I am confused at this stage. Can SUS results be used for comparison between different systems for usability? That's to say we compare the SUS scores resulted from System A and System B, and conclude that either System A is more usable than System or System B is more usable than System A.

Answer 1

Comparing usability between systems (or within different versions of the same system) is explicitly the point of the System Usability Scale.

The point of SUS scores is that they're fairly vague, and since you're supposed to be comparing generally similar systems (why would you be comparing dissimilar systems?), they can be a quick and dirty way to get some insight. It can be used to categorize your competitors into "better, on par, worse" relative to your score to help prioritize, or it can be used to compliment A/B testing for a more subjective and holistic look into usability than click accuracy/time or other objective but shallow measures.

See also this answer from the creator of SUS. Basically, SUS has stood up reasonably well under a fair amount of scrutiny (including research) due to it's popularity and cost (free). You may not be able to confidently say that system A is better because it got 1 SUS point higher than system B, but like is often the case in UX, the quick and dirty tool is more practical to use and often more insightful than very narrow, objective tests. It's not the end-all tool, but there aren't any UX tools that qualify as that, it's the nature of the beast.

Answer 2

While you might not be able to make direct comparisons between two systems, the SUS does provide a benchmark that you can use to make comparisons of the same system at different time periods. The key with SUS is that it provides an overall measure of system usability without making very specific distinction between the factors that contribute to usability, and how the weighting might be different in different types of systems. However, it doesn't mean that you can't make some assessment about the change in SUS rating over time for two different systems as an indicator of improvement in overall usability over time.