What pairs of questions test respondent's consistency?

For example, questions 2. and 3. seem to be asking the same thing, but in a way that if respondent is consistent he/she can't be agreeing with both statements at the same time: 2. I found the website unnecessarily complex 3. I thought the website was easy to use

What are the other pairs?

  • 1
    I don't agree necessarily they're exclusive. They are closely related and it would make sense that a high score in one would usually be a low score in the other but not necessarily. i.e. imagine a system where only 2 buttons are needed but for some reason the developer adds 8....it can still be very easy to use but that is needlessly complex Commented Aug 28, 2015 at 13:51

1 Answer 1


What pairs of questions test respondent's consistency?

They don't exist ;-)

The SUS wasn't designed with pairs of questions in mind. The SUS wasn't designed to test particular factors (although there has been some interesting post-hoc factor analysis of the SUS).

To quote from the original SUS paper:

SUS is a Likert scale. It is often assumed that a Likert scale is simply one based on forced-choice questions, where a statement is made and the respondent then indicates the degree of agreement or disagreement with the statement on a 5 (or 7) point scale. However, the construction of a Likert scale is somewhat more subtle than this. Whilst Likert scales are presented in this form, the statements with which the respondent indicates agreement and disagreement have to be selected carefully.

The technique used for selecting items for a Likert scale is to identify examples of things which lead to extreme expressions of the attitude being captured. For instance, if one was interested in attitudes to crimes and misdemeanours, one might use serial murder and parking offences as examples of the extreme ends of the spectrum. When these examples have been selected, then a sample of respondents is asked to give ratings to these examples across a wide pool of potential questionnaire items. For instance, respondents might be asked to respond to statements such as “hanging’s too good for them”, or “I can imagine myself doing something like this”.

Given a large pool of such statements, there will generally be some where there is a lot of agreement between respondents. In addition, some of these will be ones where the statements provoke extreme statements of agreement or disagreement among all respondents. It is these latter statements which one tries to identify for inclusion in a Likert scale, since, we would hope that, if we have selected suitable examples, there would be general agreement of extreme attitudes to them. Items where there is ambiguity are not good discriminators of attitudes. For instance, while one hopes that there would be a general, extreme disagreement that “hanging’s too good” for those who perpetrate parking offences, there may well be less agreement about applying this statement to serial killers, since opinions differ widely about the ethics and efficacy of capital punishment.

SUS was constructed using this technique. A pool of 50 potential questionnaire items was assembled. Two examples of software systems were then selected (one a linguistic tool aimed at end users, the other a tool for systems programmers) on the basis of general agreement that one was “really easy to use” and one was almost impossible to use, even for highly technically skilled users. 20 people from the office systems engineering group, with occupations ranging from secretary through to systems programmer then rated both systems against all 50 potential questionnaire items on a 5 point scale ranging from “strongly agree” to “strongly disagree”.

The items leading to the most extreme responses from the original pool were then selected. There were very close intercorrelations between all of the selected items (± 0.7 to ± 0.9). In addition, items were selected so that the common response to half of them was strong agreement, and to the other half, strong disagreement. This was done in order to prevent response biases caused by respondents not having to think about each statement; by alternating positive and negative items, the respondent has to read each statement and make an effort to think whether they agree or disagree with it.

  • In addition, items were selected so that the common response to half of them was strong agreement, and to the other half, strong disagreement. That's exactly what I'm asking about- there must be a way to check if user didn't make an effort and agreed with all or most questions- how do we test for that? I'm putting SUS answers in Excel and I'm looking for a formula that would raise a flag if it spots a response bias.
    – mmatti
    Commented Aug 31, 2015 at 9:41
  • I think you're mis-reading the intent. They're not pairs of questions that are asking the same thing in two different ways. It's alternating questions so that on one the "high" is agreement and on others the "high" is on disagreement with the idea that it would encourage the participant to read each Q in detail. This was good practice at the time SUS was created, but there's been more recent research that it can actually lead to more user errors. There's actually an 'all positive' version of SUS out there measuringu.com/positive-negative.php
    – user597
    Commented Aug 31, 2015 at 10:33
  • MeasuringU has a SUS Calculator which checks for inconsistent responses (watch their movie to see it in action). I can't think of any way to check for inconsistency other than checking if responses to closely related Q's are similar or different.
    – mmatti
    Commented Aug 31, 2015 at 12:28
  • If you're talking about the thing at 2:30 all they're doing is looking for runs of the same number across several questions (somebody went down the test and marked everything "4" — or whatever. I'd personally be a little suspicious of that as a good metric since I've seen genuine folk do that…
    – user597
    Commented Aug 31, 2015 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.