8

We are soon to be conducting an online survey, and several of the questions will attempt to measure how the user feels about a given situation.

We are using the likert scale to display the choices.

I'm trying to find out the best practice for ordering options:

I have a choice of: (example question)

The new website is better than the old one:

  • Strongly disagree
  • Disagree
  • Neither disagree or agree
  • Agree
  • Strongly Agree

or

The new website is better than the old one:

  • Strongly agree
  • Agree
  • Neither disagree or agree
  • Disagree
  • Strongly Disagree

These choices will be horizontal, in a typical "grid/matrix" style fashion.

in short - what is the general consensus - start off with a "positive" option, or start off "negatively"?

I have read some research (lost the link) which suggested starting on a positive tone effected the results slightly compared to a negative tone.

Hope its clear enough/not subjective

n.b. A/B split survey is not an option.

Improve this question
1
  • 1
    A/B split survey is not an option. - there goes my answer :) I've seen it differently, though I would claim that vertical starting with negative is most rare. It's just a feeling, though, so just as comment.
    – peterchen
    Jan 17, 2011 at 17:33

10 Answers 10

Reset to default
4

I don't have any evidence and being cynical, but I'd say it depends on reading order and what you want the answer to be. In that sense I'd agree with your lost link that depending on what tone you start off with will affect the result.

However, it could all depend on the person's attitude as well.

For example, if the person is unhappy then starting with the negative options might lead to the more negative options being selected whereas starting positively will lead them to select slightly less negative opinions. This will be because they stop when they hit the option that more or less matches their state of mind.

Equally if the person is happy then starting with positive options might lead them to select the more favourable options as they hit them first (and vice versa).

I'm not sure whether changing the order of the options is a good idea. You might end up confusing people.

Improve this answer
5

One point, slightly unrelated to your question but relevant to your task nonetheless - always write the questions in the same 'direction'. Make sure each question asks for the same type of comparison (eg 'is the X in the new better than the X in the old?')

For instance, if you write

The new site is better than the old one (agree / disagree)

Don't then write

The old site has better navigation than the new one (agree / disagree)

Keep the 'direction' of your questions the same, and users will soon catch on. This will help them complete your survey quickly, and stop intimidated / time-strapped users from leaving - improving your sample.

Improve this answer
5

Purely a personal opinion and taking a left-right reading direction I would say:

Horizontal:
 Negative -> positive

Vertical:
 Positive
^
Negative

As others have already said above, being consistent in how you phrase your questions is probably more important.

Improve this answer
4

I agree with Bobby. The language makes a huge difference. I would be curious to see if you got different results with different wording. I think the "neutral in the middle" pattern is pretty consistent, and it may not matter if positive or negative is the left-most/first option, as long as you make the scale clear. In a survey, I don't want to waste time figuring out if positive is left or right. I don't care which side it's on, as long as I know which side it's on.

What about rating more qualitatively, and asking "Which site did you prefer?" old <-----> new. The downside there is people like the things they are familiar with.

Improve this answer
4

The answer to nearly all survey style questions of this nature is merely this: be consistent with what you are comparing your results to. If you are running this survey over time to determine a baseline and measure changes, then just keep your questions and responses consistent over time.

There will always be some bias based on how you ask the questions, but you'll go nuts trying to perfect every question because for the most part these sources of bias are very context-dependent and have only been quantified in limited situations.

Now, you probably notice that my feedback doesn't really apply for the statement "The new website is better than the old one". Unless you have two separate re-designs you are testing simultaneously with a survey, you'll never have a comparable group for comparison. In my opinion, that means the question is not going to provide really meaningful results. For a survey of a subjective topic, the scales are always going to be arbitrary. Especially for a hard question asking them to compare how much they like the current website compared to the previous one.

What I would do is come up with a question that measures some aspect of their experience today, run that survey today, and run the same survey after you launch the re-design. See if the scores go up or down, try to identify sub-groups that may be affected differently. For many websites, you would ask what their purpose in visiting the website that day, and then ask a followup question of whether they were able to accomplish that task.

Improve this answer
3

I doubt you'll like this answer but, in my opinion, the wording of the question will have far greater effect than the order of the rating options. Compare "the new website is better than the old one" with "the old website is better than the new one".

Improve this answer
3

One thought is how many people think from low --> high OR form bad --> good OR negative --> positive.

I have no evidence for this just experience, I even have once where I had colored the scale from green to red, but still people voted as if it was from red (bad) to green (good).

Improve this answer
3

Although I'm missing references to the actual paper, I remember reading about some research about selection tendencies and biasing on linearly presented choices. The conclusion was roughly saying that people were slightly inclined to pick the first choice, and slightly less inclined than the first but still more than the rest to pick the last choice. The distribution of inclination among the "middle" choices remained statistically insignificant.

If that research holds true in this case, you may get a slight bias towards the beginning of the scale, whichever way you pick it.

Improve this answer
0

If the survey is being taken on a mobile device, would it be wiser to have the positive answers near the bottom, as they are the easiest to reach with the thumb? So negative to positive order.

Improve this answer
0

I think there's a better question to be asked. The current question isn't really a question, it's a statement and the user has to decide how much they agree with the statement.

If you're trying to discover user's feelings, ask about their feelings directly. Look at how the current question makes a statement then asks how the user how they feel about the statement.

As far as the negative or positive first issue, if you build the negative and positive into the question, it can take a lot of the weight off of which is first. There's a lot less worrying about being right or wrong when describing your feelings as opposed to trying to assess websites.

Here's an example that tries to do this:

How do you feel about our new website?

  • I love it!
  • I like it
  • Didn't even notice
  • I don't like it
  • I want the old site back!
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.