Using SUS to measure usability of an existing system and compare with a redesigned system

Question

I will be undertaking a whole-scale redesign of an existing system. I'd like to use SUS ("System usability scale") questionnaires to measure improvements between iterations.

As a baseline, I want to measure the current system using SUS but one of the questions don't make sense and all of the questions will need to be rephrased into the present tense.

The question that doesn't make sense is the first question ("I think that I would like to use this system frequently.") since the system is a workplace system that the users are required to use constantly.

I've rephrased the first question as, "I like to use [system name] frequently." I've put all other questions into the present tense e.g. "I find [system name] unnecessarily complex."

Is that OK? Will this baseline measure still be comparable with later results (using an unmodified SUS questionnaire) as we iterate through redesigns?

Answer 1

You seem to be implying that you will be surveying current users for the current system, and this is likely to be problematic beyond the mere appropriateness of the survey questions.

A basic principle in research (which is what you are doing) is that, for a valid comparison of two things, everything else has to be the same other than what you are comparing.

In this case, you would want the same kind of users to test each system. The current user base will be experienced users with respect to the old system but brand new users with respect to the new system. The way they approach the two systems won't be at all equivalent, and thus, neither will their answers to the survey. Further, their knowledge of the old system will inform the way they approach the new one, which also invalidates the comparison.

Some of the potential consequences of this problem:

• People may prefer the old system simply because they are resistant to change.
• People may repeatedly make mistakes with the new system because they try to do things the "old way" and press the wrong buttons, etc.
• People may think the old system is a pile of garbage, preferring the new one simply because it is new and shiny--even if the new one is not really usable.

You will probably end up measuring things like people's resistance to change or how they feel about their job for the last five years, rather than comparing the performance of the two systems.

If you truly want to compare the two systems, the only real way for the comparison to be valid is to have completely new users test both systems.

If this is impractical (as is likely), a fallback option is to have existing users test the new system only. You will still have some of the problems of bias, but there are fewer consequences when you aren't making a direct comparison to the old system. And you will gather lots of useful information this way.

Answer 2

I fully agree with dan's answer, just to answer your questions though.

I would be hesitant to change the wording of the survey from test to test. We'll be introducing another source of variance from the different tenses. It might make sense to preface all the questions in the survey with "Imagine you are a new user of this system" to avoid the "this is my current system" confusion and not affect future iteration tests.

The question that doesn't make sense is the first question ("I think that I would like to use this system frequently.") since the system is a workplace system that the users are required to use constantly. I've rephrased the first question as, "I like to use [system name] frequently."

You can reword the question. Again, it should be done consistently across all your surveys. Perhaps "I think that I would want to use this system frequently." might fit better without altering the meaning too much. If you're publishing these results, you'll need to mention you've used a modified SUS questionnaire.