Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

I'm preparing a UX survey which will use a Likert scale.

Personally I'd rather use the same labels configuration for all the questions/items such as :

1. Strongly disagree
2. Disagree
3. Neither agree nor disagree
4. Agree
5. Strongly agree

Now, since it's a collaborative work, some of my colleagues have added questions that use different labels (for tasks frequency for example) and sometimes in a different order.

My gut feeling is that we should edit the questions so that all the choices labels are exactly the same on the whole survey.

Or do I care too much?

share|improve this question

3 Answers 3

Having the same order do matter. It should be the same scale and order across the survey. Otherwise you can't be sure that your user understood what they where answering.

I'd consider the use of an even number answer option in a 4-point Likert scale forcing users to take an active position on the questions asked. Having an "neither/nor" option somewhere in the middle may be used as "no sure" or "no opinion" answer. On the 4-point scale you can add one of these option making it more clear what the user ment. That way your survey result will be more valid.

5-point or 4-point is you decision, but make a decision of which one to use.

share|improve this answer

No, you don't have to use the same answer scale.

In a survey, you are measuring different variables. Each variable is measured by a number of metrics, e.g. usability can be measured by the metrics learnability, ease of use and so on. Different metrics can, and will, have different units. This is not so different than in physics, where time is measured in seconds and distance in meters.

As you are working with self-reported variables, you have a bit of a leeway as to how to get the information on some metrics. You can ask "How much difficulty did you have learning to use our product", and use a scale ranging from "A lot" to "None at all", or you can say "The product was difficult to learn", and use a scale ranging from "Strongly disagree" to "Strongly agree". As long as you can make the two scales equivalent*, there is no difference in the final information. It is like choosing between measuring distance in meters or yards, you get the same information out of it.

You should measure the same metric in the same way. This means, if you make two studies one year apart, the comparability of the results between the studies will be reduced a lot if you choose one scale for the first study and the other for the second study. But between variables, there is no such issue. There is no inherent comparability between task frequency (which is measured in repetitions per time unit) and e.g. learnability (which is measured in a unitless ordinal measure), or even between different metrics measured with a unitless ordinal measure. So there is nothing wrong with measuring the different metrics with different scales, you are losing nothing in terms of comparability. From the point of view of the evaluation, there is no reason at all to use the same scales.

Then there is the question of the users. Are you going to confuse them if you use different scales? I don't think that this will happen. I haven't seen it happen in my own studies. The world is also full of other examples. My favorite supermarket sells kiwifruit by the piece, oranges by the kilogram, eggs by the six-piece package, and cheese packaged in arbitrary-size packs. None of the customers gets confused by this. We are accustomed to using different units for different things all the time. Similar for your questionnaire - you are measuring different things, so measuring them with a different scale won't confuse anybody.

[*] On the matter of making scales equivalent: Agree-Disagree scales are sometimes less expressive than more direct scales. If you are measuring the user's opinion of the current level of some attribute of your product, they are bad. For example, asking "How do you feel about the length of our newsletter" can have a scale ranging from "too long" to "too short", which is more expressive than saying "I feel that your newsletter is just the right length" and then having an agree-disagree scale.

share|improve this answer
1  
I bet that customers do get confused by the variety of measurements. In some shops, you still have to weigh your products. Sometimes I find myself in the situation where I have to check three times whether I have to use the scale or if its per piece. In the more modern shops, where the scales are integrated in the cashing machines, your point is valid. Since I don't have to care I wont be confused. –  L. Möller Dec 17 '13 at 12:09
1  
@L.Möller you said it yourself - the problem with this strategy that you "have to check three times whether I have to use the scale". If the supermarket had tattoeed "please use scale" on each banana, there would be no problem :) Now, this strategy may be hard to implement for a supermarket. But on a questionnaire, it is easy to label every single scale, so the user knows at a glance what their answers mean. –  Rumi P. Dec 19 '13 at 12:01

Information wise, having a scale of values that is well connected to the question being answered would be ideal. For example, "How often do you exercise?" having "Daily, Twice a week, Twice a month, Rarely" as options would make sense to the reader. Avoid rewriting the questions for the sake of using the same options everywhere.

Usability wise, maintaining a consistent order of the information is important. Users get used to this quickly as they get through the questions. Depending on how the questions are phrased and list of options given, users would assume that the first one is always 'the maximum' (Strongly Agree / Daily / Very Difficult) and the last is always 'the minimum' (Strongly Disagree/ Rarely / Very Easy) - before reading the actual information. It can be the reverse, but should be consistent across the questions.

People usually decide on their answer from the question itself, before reading the options, and then choose an answer from the list that matches the most to the one they had in mind.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.