I'm trying to evaluate the validity of the SUS responses- I think it's also called internal reliability or Response bias. How is this done for SUS?
6
-
I really think this is a question for stats.stackexchange.com. It has little to do with user experience.– IzhakiAug 17, 2015 at 22:24
-
I'm voting to close this question as off-topic because it belongs to Cross Validated– DevinAug 18, 2015 at 1:20
-
It depends- if it's just a calculation, then yes it's more of a stats question. But if it requires to know which pairs of SUS questions ask for the same thing it's more UX.– mmattiAug 18, 2015 at 5:35
-
@mmatti It's currently unclear exactly what you're asking, and the focus of your question will determine whether it's on topic or not. In the mean time, I'm closing it, but if you improve the question, you can ask to have it reopened.– JohnGB ♦Aug 18, 2015 at 9:02
-
It is totally related to hci!!! Not offtopic to me. Maybe we need more context but it is interesting– maiaAug 19, 2015 at 17:42
|
Show 1 more comment
1 Answer
When analyzing survey data you need to separate validity from reliability.
- Validity is a measurement of how correct the answer is.
- Reliability is a measure of how consistent the answer is.
Here's a graphical representation of the concept (from Wikipedia):
In general, SUS has been found to be highly reliable and there is some strong support that it produces valid conclusions [1].
But you are probably more interested in determining if your survey results are valid for your problem. To answer this, my advice is to find some other measure of usability to correlate your results with (this is referred to as "convergent validity"). For instance, you could compare the results of a similar survey or expert evaluation of the interface with the SUS scores. Or you could compare task completion time (or some other quantitative measure of performance) of a user with previously gathered SUS scores, if they correlate then SUS has "predictive validity", which would be pretty convincing.
If you are interested in reliability, I suggest you use Chronbach's alpha. It is a combined measure of all correlations between questions and essentially means how likely is it that respondents answer questions in a similar way. Alternatively, you could rerun the test with the same participants and see if you get similar answers.
[1] Bangor, A., Kortum, P. T., Miller, J. T.: An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction. 24, 574--594 (2008)
