11

What major 'competitors' exist for the SUS survey?

Is there anything picking up a significant degree of acceptance as a standard to the extent that SUS has?

In particular...is there anything which produces somewhat similarly reliable results with fewer questions?

5
  • SUS? What does that mean?
    – UXerUIer
    Commented Feb 11, 2015 at 13:45
  • System Usability Scale. I'll embed a link in the question for context for those unfamiliar with the term. OP - If you're using acronyms in questions you shouldn't assume everyone knows what you are referring to.
    – JonW
    Commented Feb 11, 2015 at 13:56
  • You are asking 3 separate questions here; I'm not sure which you want answered. Commented Feb 11, 2015 at 15:22
  • The same question broken up I'd say. As one: Are there any accepted, reliable competitors to SUS, with particular interest paid to shorter surveys? Commented Feb 12, 2015 at 12:42
  • I really wonder how usable these scores are?
    – Praasshant
    Commented Apr 23, 2015 at 16:08

7 Answers 7

7

Jim Lewis wrote an article in 2012 for Measuring Usability

The article explains the direct correlation found between SUS scores and Net Promoter Scores (NPS). They found that it was possible to predict NPS scores based on SUS scores, which means the inverse is also true.

The Net Promoter Score is a widely used survey which consists of only one question. That's a lot less than the 10 questions in the SUS survey :)

The correlation between the two scores was drawn from over 2200 users across 81 different companies and products including rental car companies, financial applications and websites like Amazon.com.

The relationship published is:

LTR = 1.33 + 0.08(SUS)
3

There are the SUMI and WAMMI from University College Cork (http://www.ucc.ie/hfrg/questionnaires/index.html). They require a fee for use and are longer than the SUS.

3

In Germany these two questionaires are used (and I think known in usability circles):

  • AttrakDif
  • UEQ

I'll cite from my answer here: https://ux.stackexchange.com/a/72905/3311 :

AttrakDif

The AttrakDiffTM questionnaire by Hassenzahl et al. (2003), developed together with User Interface Design GmbH, measures subjective assessments concerning pragmatic and hedonic qualities and the attractiveness of interactive products.

The questionnaire has the format of a semantic differential and uses 28 seven-step items on 4 dimensions which poles are opposite adjectives (e.g. complicated - simple, cheap - premium, conservative - innovative, ugly - attractive). Source: http://www.qu.tu-berlin.de/menue/forschung/laufende_projekte/joyofuse/joy_of_use/joy_of_use/measurement_methods/attrakdiff/

UEQ - User Experience Questionaire

The User Experience Questionnaire (UEQ) allows a quick assessment of the user experience of interactive products. The format of the questionnaire supports users to immediately express feelings, impressions, and attitudes that arise when they use a product. Source: http://www.ueq-online.org/index.php/what-is-the-user-experience-questionnaire/

2

Regarding the question, "In particular...is there anything which produces somewhat similarly reliable results with fewer questions?"

You should take a look at the published research on the Usability Metric for User Experience, published by Finstad (2010). The UMUX has four items and has typically been found to have desirable psychometric properties and to have scores that correlate highly with the SUS and, in some studies, have also produced scores that track very closely with SUS scores in magnitude.

In 2013, Lewis, Utesch, and Maher took two of the UMUX items to produce the UMUX-LITE. According to the abstract of that CHI paper, "In this paper we present the UMUX-LITE, a two-item questionnaire based on the Usability Metric for User Experience (UMUX). The UMUX-LITE items are “This system’s capabilities meet my requirements” and “This system is easy to use.” Data from two independent surveys demonstrated adequate psychometric quality of the questionnaire. Estimates of reliability were .82 and .83 – excellent for a two-item instrument. Concurrent validity was also high, with significant correlation with the SUS (.81, .81) and with likelihood-to-recommend (LTR) scores (.74, .73). The scores were sensitive to respondents’ frequency-of-use. UMUX-LITE score means were slightly lower than those for the SUS, but easily adjusted using linear regression to match the SUS scores. Due to its parsimony (two items), reliability, validity, structural basis (usefulness and usability) and, after applying the corrective regression formula, its correspondence to SUS scores, the UMUX-LITE appears to be a promising alternative to the SUS when it is not desirable to use a 10-item instrument."

There have been additional studies of the UMUX-LITE since then. Here is a list of papers to read on the topic:

Borsci, S., Federici, S., Bacci, S., Gnaldi, M., & Bartolucci, F. (2015). Assessing user satisfaction in the era of user experience: Comparison of the SUS, UMUX and UMUX-LITE as a function of product experience. International Journal of Human-Computer Interaction, 31, 484-495.

Finstad, K. (2010). The usability metric for user experience. Interacting with Computers, 22, 323-327.

Finstad, K. (2013). Response to commentaries on "The Usability Metric for User Experience". Interacting with Computers, 25, 327-330.

Lewis, J. R. (2013). Critical review of "The Usability Metric for User Experience". Interacting with Computers, 25, 320-324.

Lewis, J. R., Utesch, B. S., & Maher, D. E. (2013). UMUX-LITE – When there’s no time for the SUS. In Proceedings of CHI 2013 (pp. 2099-2102). Paris, France: Association for Computing Machinery.

Lewis, J. R., Utesch, B. S., & Maher, D. E. (2015). Measuring perceived usability: The SUS, UMUX-LITE, and AltUsability. International Journal of Human-Computer Interaction, 31, 496-505.

0

Promoters − Detractors

I'm not sure how you define “competitors”, but ForeSee has developed a claimed successor to Net Promoter Score: WoMI > Word of Mouth Index. It’s just a trademark on a basic premise some researchers have been using for a while (ForeSee likes to bill for simple things).

WoMI alters the simplistic NPS formula with only one extra question on the 0–10 scale: likelihood to discourage (LTD) in addition to likelihood to recommend (LTR). You then factor together the strong responders from each (9s and 10s), to find your true score.

It’s not as detailed as SUS, but as you suggest, 10 questions isn’t always ideal. I like the fact that with a simple promotersdetractors calculation I can get the fundamental score I'm after. I find that it’s the lightest weight way to get a realistic indication of feature success.

0

Jeff Sauro published a paper in the Journal of Usability Studies on a new survey he has created. In the paper, he has a good summary table of the surveys currently in use. SUPR-Q: A Comprehensive Measure of the Quality of the Website User Experience

0

There are also these:

The SUPR-Q, or Standardized User Experience Percentile Rank Questionnaire; this is a product of Jeff Sauro's Measuring U group which (IHMO) is one of the gold standards for thought leadership around data and metrics. It used to be 13 items, it's now down to 8, and it is used by a LOT of very high profile companies. It's not cheap, but could be worth exploring https://measuringu.com/product/suprq/

UMUX, or Usability Metrics for User Experience, is much shorter than the SUS for more time-constrained engagements, only 4 questions on a 7-point scale. It is meant to capture a lot of what the SUS captures in a smaller space... as Measuring U puts it "It is intended to be similar to the SUS but is shorter and targeted toward the ISO 9241 definition of usability (effectiveness, efficiency, and satisfaction)." There is even a 2-question version for those with very little user attention. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.872.6330&rep=rep1&type=pdf

QUIS (Questionnaire for User Interaction Satisfaction)is meant to be a sort of general evaluation tool for any interactive computer system. I have not ever even SEEN this one, http://www.cs.umd.edu/hcil/quis/

The somewhat unfortunately named PSSUQ (Post Study System Usability Questionnaire) and CSUQ (Computer System Usability Questionnaire), which are almost as widely used as the SUS. Here is a good article on Measuring U that describes them: https://measuringu.com/pssuq/ https://smile.amazon.com/USING-PSSUQ-CSUQ-Experience-Research/dp/1733339205/

If you get desperate, I also developed my own version with a colleague, which we call the Akamai Usability Dyads. It is a sort of hybrid of SUS and MS Product Reaction Cards - vaguely akin to the AttrakDif - and is geared more specifically towards evaluating enterprise level SaaS software. It has been vetted by several UX research and data experts and has an evaluation tool built in... some day I may actually publish something about it.

enter image description here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.