2

I'm doing User Testing and collecting feedback from users at the end of the task.

Some questions require the user to give a numerical rating on how easy or difficult the task was, would they recommend the product to others, and so on... (1-Strongly disagree, 7-Strongly agree)

I can choose to display this in various ways,

Numerical
1, 2, 3, 4, 5, 6, 7

Star Rating
☆☆☆☆☆☆☆

Emoticons
enter image description here

My Question:

I'm wondering if there are studies or research I can read, about how the format of the question affects the results. Things like:

  1. Does using emoticons result in a overall higher rating compared to numbers?
  2. Does putting 1 as the Positive option (as opposed to the Negative option) result in more positive feedback?
  3. How does the scale skew the result? e.g. 1-5 VS 1-7
  4. Should I start with a value selected by default? Which value should it be?
  5. How should I format the question if I want the user to rate more positively?

Appreciate if someone can point me to resources.
Thank you!

3
  • 1
    "if I want the user to rate more positively" - What is the purpose to ask users for feedback, if you want manipulate it? You will have positive feedback for bad product/features. Then you or your customer will invest money into a bad product and will loose these money, because users prefer other product. But you will not know that you product is bad, because you manipulated feedback and got better results.
    – mentallurg
    Apr 21, 2023 at 5:13
  • Numbers and scales can be compared easily. Whereas emoticons are almost not comparable. Take a single emoticons without any context and ask people how much positive or negative this particular emoticon is. For 4-5 you can get more or less similar answers, but the more emoticons you use, the more random will opinions look. For instance, your picture with 7 emoticons shows already problems: 1) The 5th emoticon looks more negative than the 4th one. Decisions make on too many emoticons can lead to wrong results.
    – mentallurg
    Apr 21, 2023 at 5:30
  • ... 2) Color perception is very individual and can lead to wrong results. For instance, blue colour on your picture may be felt much colder than yellow, and the 3rd emoticon can be percepted as more positive than the 7th one.
    – mentallurg
    Apr 21, 2023 at 5:32

1 Answer 1

1

I can tell you from my own experience with data from millions of users, that, for the same experience or product, any rating will vary wildly in interpretation between different cultural groups. As a general example, 4 stars in Germany is roughly equivalent to 4.5 stars from Spain, and the evidence is that it's not based on the experience, but rather on what 4 stars or 5 stars means to them. While I've been specific to cultural differences, the differences between people (even within a given culture) are often more pronounced that the differences between cultures, it's just harder to quantify, hence the culture reference.

So, the problem often is that when you present a purely numerical or star based response, there is no common idea of what each rating represents in terms of experience. That is largely improved by using an emoji rating that ties into something more common in the human experience, which is how the interaction / experience made you feel. As a result (and this is confirmed by my own experience), you get more consistent and accurate feedback from emoji based rating systems than stars or numerical rating.

The exception to this is when the rating criteria are clearly understood and the rating is happening by an expert user. For example, if I had a welding inspector inspecting gas pipeline welds, it's likely that each numerical rating would have a specific, well defined, and commonly understood meaning, and for that an emoji system would be inappropriate.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.