What to do if SUS scores contradict qualitative feedback?

Question

TL;DR: qualitative data collected in a usability experiment seems to contradict the quantitative results of the SUS questionnaire. How can this discrepancy be reconciled?

The following experiment is conducted to evaluate the usability of a web-interface:

1. Observe participants as they think aloud while using the interface to accomplish 8 tasks (the task order is randomized, this takes around 30 minutes)
2. Give them a SUS form to fill out
3. After they completed the survey, ask several follow-up questions to get more feedback (another 30 minutes)

So far, the experiment was conducted with 5 participants, then the UI was adjusted to address the found issues. A second round of 5 participants were then invited to go through the same steps.

It is planned to perform another round, with at least 5 participants (to obtain a sufficiently large sample). The current results are summarized below:

You can see that the v2 score is lower than v1.

These findings are puzzling, because:

• the qualitative feedback I got from participants was more positive in v2

• the changes between v1 and v2 were not ground-breaking, e.g.:

  • added tooltips to widgets
  • increased the contrast to make the active tab more prominent
  • changed wording to avoid technical jargon
  • shortened text

• nevertheless, these tweaks did polish the "rough edges" of v1, as it was clear from the observations that there was less friction while participants used the site

In other words, the changes were small incremental steps that should have yielded small improvements. The qualitative results match the expectations, while the quantitative data do not.

Since the overall average of 69 falls in line with the average SUS score of 68, it seems that nothing unusual has happened and we're testing "just an average interface". However, I am not sure how to reconcile the fact that the numbers contradict the humane feedback.

Nielsen says that qualitative feedback is more valuable and numbers can lead you astray. On the other hand, Sauro says that they do report SUS scores based on a sample of 5 users (as well as looks at the history of sample sizes, concluding that a minimum of 5 is reasonable).

At the same time, a t-test says that the differences between the scores of v1 and v2 are not statistically significant.

How could one make sense of these results?

---

Thank you all for your comments, answers, and time. Although there is only one accepted answer, all the input is helpful. It enabled me to take a sober look at the data, and reduce the "jumptoconclusionness" factor to a lower level.

A note for future archaeologists: the question was edited to include details and statistics mentioned in the comments. It might help to look at the edit history to see the starting point and understand how it ended up like this.

Answer 1

How can this discrepancy be reconciled?

You have divergent results because the number of participants is small and not representative. There is no randomization or blinding to prevent bias. You're also not calculating the relevant stats. (What are the standard deviation, margin of error, confidence intervals, odds ratios, p values, etc?)

Further, you appear to be doing iterative design, not "experiments". There is nothing wrong with iterative design, but the data you collect are likely irrelevant beyond the current design. They cannot be used to meaningfully compare designs against each other. Even if they could, there aren't enough participants to measure the effect of small changes. But you don't need large numbers of users for iterative design. Just enough to identify improvements for the next iteration.

In an experiment, you'd have multiple designs A/B/C... tested in parallel. Participants would be randomized to the designs (as well as task order). Experimenters would not know which design individual participants were using. Experimenters would not observe participants directly. Experimenters would pre-decide what statistical tests are appropriate. They would not begin processing data until after it had all been collected. Etc. If you were testing drugs, your methodology (as well as insufficient participants) would likely prevent FDA approval.

How could one make sense of these results?

You did a t-test and found no significant difference. The "study" is likely underpowered with only five subjects in each group. Even if you had enough numbers to demonstrate significance, the study needs to be redesigned, and the survey has to be checked for reliability and validity.

The System Usability Scale (SUS) is described by its original developer as "quick and dirty". It appears to have been validated as a global assessment, but it's probably not appropriate for comparison. Imagine there were something known as Global Assessment of Functioning that physicians used to evaluate health. Is someone with condition A and GAF 85 "healthier" than someone with condition B and GAF of 80? Does it even make sense to compare A and B this way?

Even if these problems were all addressed, you are still doing iterative design. I would expect differences between successive iterations to be non-significant. Suppose you were testing drugs. Would you expect significantly different results between 100mg and 101mg doses? What about 101mg and 102mg? Etc. (How massive would n need to be to detect such minute differences?)

What to do... ?

Understand that iterative design is not experimentation. The value of small usability reviews is to screen for problems, not confirm success or produce stats.

Stop collecting (or "misusing") quantitative data when you know you won't have the numbers to demonstrate significance. Stop having "expectations", as it is a source of bias that can lead you astray. Redesign experiments to reduce bias.

... it seems the confidence intervals are so wide, that the intermediate results I got should not be a reason of concern.

That is as "expected".

Answer 2

How to reconcile the discrepancy? That I can't tell but here is why it might have happened.

The "5 users will find everything wrong with your system" refers to usability problems test subjects will find in your usability tests. Sauro has a great article that goes quite deep in to this "5 users is enough" thingie.

The five user number comes from the number of users you would need to detect approximately 85% of the problems in an interface, given that the probability a user would encounter a problem is about 31%.

Jeff Sauro - Why you only need to test with five users (explained)

SUS on the other hand needs bigger sample size to yield meaningful results. On their 2010 paper A Comparison of Questionnaires for Assessing Website Usability Tullis & Stetson came to a conclusion that you need a sample size of 12 or more for SUS to yield a 100 % accurate analysis of whatever you are analysing.

As one would expect, the accuracy of the analysis increases as the sample size gets larger. With a sample size of only 6, all of the questionnaires yield accuracy of only 30-40%, meaning that 60-70% of the time, at that sample size, you would fail to find a significant difference between the two sites.

So with sample size of 5 you'll get something meaningful in less than 30 % of the time.

For example, SUS jumps up to about 75% accuracy at a sample size of 8, while the others stay down in the 40-55% range. It’s also interesting to note that most of the questionnaires appear to reach an asymptote at a sample size of 12.

Then again, what is an adequate sample size for qualitative analysis. I haven't done any qualitative analysis in the UX field so I can't tell what would be the sufficient sample size. This I found on internet:

Qualitative sample sizes should be large enough to obtain enough data to sufficiently describe the phenomenon of interest and address the research questions.

Oh, really.

For an ethnography, Morse (1994) suggested approximately 30 – 50 participants. For grounded theory, Morse (1994) suggested 30 – 50 interviews, while Creswell (1998) suggested only 20 – 30. For phenomenological studies, Creswell (1998) recommends 5 – 25 and Morse (1994) suggests at least six.

Qualitative Sample Size

Answer 3

Look at your confidence intervals: the "real" score for v1 is somewhere between 58 and 88, while that score for v2 is somewhere between 51 and 79. Most notably, the confidence interval for each score contains the mean for the other score.

What this is telling you is that your sample size is too small. Based on the data you've collected so far, you cannot tell anything about the relative merits of the two versions of the user interface. There are various statistical tests you can use to quantify just how thoroughly you don't know anything, but checking for overlapping confidence intervals is fast and reasonably accurate.

Answer 4

I think the most important thing is to realize that using a combination of qualitative and quantitative analysis methods is going to give you the most complete best picture of what the users are thinking and doing rather than relying on one or the other (which as you can see led to different conclusions). The idea is to use quantitative analysis when you have larger datasets to look for large trends or patterns, and use qualitative methods to drill in on the specific issues. It is also possible to combine various observations in qualitative methods and look for evidence/support of a general trend in the quantitative analysis.

Something that I think is missing in the analysis (in additional to what has already been raised in the other answers) is that there are potentially other contextual information that you haven't used to split some of the responses.

For example, I can imagine that there are different types of user groups mixed together in the analysis and that some of the users have a particular mental model or preferences that influence the more subjective elements of the SUS questionnaire.

However, when you look at the quantitative data (which should probably also be based on some of the analytics collected while they are completing the task rather than just observations) you will probably find less bias or subjectivity as you build up the number of users/participants.

Answer 5

Three scores are higher and two are lower at first. One outlier (Voting 37 instead of 90) would be enough for that. Not to mention that the very close totals mean there is not a lot of difference. Could be that the mood/whether or your participants were different.

Though there might actually be another issue. If we take the results at face value (even if the sample size is too small), the newer version seems to lack the ability to make people love it.

For instance, while tool tips can be helpful and actually make someone find an option faster, I personally find them very annoying, usually. They just distract unnecessarily, especially if the option would be quite easy to find without it. And they make one feel pushed in a certain direction - similar to bad advertisement.

Also, if you omitted information which would be expected, or dumbed down things too much, people would not always be happy about that - especially if they feel that usefulness is reduced. Some people might also dislike the lack of jargon if it belongs there. Or the new colors. And so on.

As I don't know your ui in detail, I can't say more. But if you look at both versions side by side, you might see some of the annoying parts yourself, and can try to find alternatives - small hover texts instead of tool tips one has to acknowledge, for instance.