Does the format of the Feedback Question (Numbers vs Stars vs Emoticons) affect the result?

Question

I'm doing User Testing and collecting feedback from users at the end of the task.

Some questions require the user to give a numerical rating on how easy or difficult the task was, would they recommend the product to others, and so on... (1-Strongly disagree, 7-Strongly agree)

I can choose to display this in various ways,

Numerical
1, 2, 3, 4, 5, 6, 7

Star Rating
☆☆☆☆☆☆☆

Emoticons

My Question:

I'm wondering if there are studies or research I can read, about how the format of the question affects the results. Things like:

1. Does using emoticons result in a overall higher rating compared to numbers?
2. Does putting 1 as the Positive option (as opposed to the Negative option) result in more positive feedback?
3. How does the scale skew the result? e.g. 1-5 VS 1-7
4. Should I start with a value selected by default? Which value should it be?
5. How should I format the question if I want the user to rate more positively?

Appreciate if someone can point me to resources.
Thank you!

Answer 1

I can tell you from my own experience with data from millions of users, that, for the same experience or product, any rating will vary wildly in interpretation between different cultural groups. As a general example, 4 stars in Germany is roughly equivalent to 4.5 stars from Spain, and the evidence is that it's not based on the experience, but rather on what 4 stars or 5 stars means to them. While I've been specific to cultural differences, the differences between people (even within a given culture) are often more pronounced that the differences between cultures, it's just harder to quantify, hence the culture reference.

So, the problem often is that when you present a purely numerical or star based response, there is no common idea of what each rating represents in terms of experience. That is largely improved by using an emoji rating that ties into something more common in the human experience, which is how the interaction / experience made you feel. As a result (and this is confirmed by my own experience), you get more consistent and accurate feedback from emoji based rating systems than stars or numerical rating.

The exception to this is when the rating criteria are clearly understood and the rating is happening by an expert user. For example, if I had a welding inspector inspecting gas pipeline welds, it's likely that each numerical rating would have a specific, well defined, and commonly understood meaning, and for that an emoji system would be inappropriate.