4

I'm trying to evaluate the validity of the SUS responses- I think it's also called internal reliability or Response bias. How is this done for SUS?

  • I really think this is a question for stats.stackexchange.com. It has little to do with user experience. – Izhaki Aug 17 '15 at 22:24
  • I'm voting to close this question as off-topic because it belongs to Cross Validated – Devin Aug 18 '15 at 1:20
  • It depends- if it's just a calculation, then yes it's more of a stats question. But if it requires to know which pairs of SUS questions ask for the same thing it's more UX. – mmatti Aug 18 '15 at 5:35
  • @mmatti It's currently unclear exactly what you're asking, and the focus of your question will determine whether it's on topic or not. In the mean time, I'm closing it, but if you improve the question, you can ask to have it reopened. – JohnGB Aug 18 '15 at 9:02
  • It is totally related to hci!!! Not offtopic to me. Maybe we need more context but it is interesting – maia Aug 19 '15 at 17:42
3

When analyzing survey data you need to separate validity from reliability.

  • Validity is a measurement of how correct the answer is.
  • Reliability is a measure of how consistent the answer is.

Here's a graphical representation of the concept (from Wikipedia):

In general, SUS has been found to be highly reliable and there is some strong support that it produces valid conclusions [1].

But you are probably more interested in determining if your survey results are valid for your problem. To answer this, my advice is to find some other measure of usability to correlate your results with (this is referred to as "convergent validity"). For instance, you could compare the results of a similar survey or expert evaluation of the interface with the SUS scores. Or you could compare task completion time (or some other quantitative measure of performance) of a user with previously gathered SUS scores, if they correlate then SUS has "predictive validity", which would be pretty convincing.

If you are interested in reliability, I suggest you use Chronbach's alpha. It is a combined measure of all correlations between questions and essentially means how likely is it that respondents answer questions in a similar way. Alternatively, you could rerun the test with the same participants and see if you get similar answers.

[1] Bangor, A., Kortum, P. T., Miller, J. T.: An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction. 24, 574--594 (2008)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.