Factors that determine suitability of rating system/scale for user satisfaction (for an app)

Question

We want to ask our users whether they are satisfied with our app or not. For this purpose we will display a pop-up upon entering the app with 1 question. how satisfied are you with our app. I'm trying to convince the other stakeholder we should use a 5 scale answer and not a 2. the 2 scale answer will have a thumb up and thumb down will the 5 scale answer will have the star rating.

1. which of the 2 is more effective? my opinion is clear about that
2. can you give a reference for research/use case in your answer please

Thanks.

according to this the 2 is less preferred. https://conversionxl.com/blog/survey-response-scales/

Answer 1

The question you will ask has a lot of effect on what type of scale you use. Most of my answer comes from research into surveys, so for instance when you use a questionnaire to know the market impact of a certain product. (When you are walking in a store you sometimes have people asking questions about a certain 'soon to be released' product.)

When to use a 5 point scale (Likert scale)

A 5 or 7 point scale is best used when you want to measure latent constructs. In short, latent constructs are questions about: Opinions / feelings etc. These are unobservable individual characteristics.

When to use Yes/No questions

The Likert scale is best used when you want to ask latent constructs, so a yes/no question is best used when you can confirm an answer. 'Are you married' 'Do you have kids'.

What should you use now

Given that you want to ask if people are satisfied with the application, the best course of action will be a Likert scale. Finally, if you want to check the reliability of your 5 / 7 point Likert scale you can use Cronbach's alpha

I could not find any 'research' specifically on applications, however I did find something regarding the 5/7 point Likert scale that you can use to convince your stakeholders.

In current practice, most rating scales, including Likert-type scales and other attitude and opinion measures, contain either five or seven response categories (Bearden, Netmeyer, & Mobley, 1993; Peter, 1979; Shaw & Wright, 1967).

My personal opinion is that you should preferably not use a popup to ask this question, having a UX test with the target group in a controlled environment will give better answers then a (usually annoying) popup. If you do want to implement the question I suggest you do it at the end of the funnel, so after they used it to do a certain task and not at the startup.

Answer 2

If you want to know which method would let you know how users feel about the app: The 2 scale answer gives a more accurate interpretation of the users satisfaction because a user only selects a up or down type vote.

A 5 scale measurement has a middle value, the middle value gives a ‘neither here nor there’/neutral type answer which tells you nothing about the users preference. It tells you they like it and dislike it but not why, there is a chance for a lot of neutral opinions/selections which will give you little impression as to if the user likes it or not.

So you if you want to do more than a 2 scale answer go with an even number (like 6 or 4) because it has no middle value and will give you more definitive data by forcing the users to give an opinion on either end of the scale.

Take a look at this: https://www.tandfonline.com/doi/abs/10.1080/00221309.2014.994590?src=recsys&journalCode=vgen20

I would try collect some form of qualitative data too to try and learn why they selected what they did rather than to just get rid of the pop up.

Answer 3

The most interesting example that came to mind was the well publicized (but not so well analyzed) change in the Netflix rating system from the 5 star system to a more facebook-like thumbs up/down system.

However, I couldn't find a lot of articles that talk about the rationale for this change that goes into more detail (even Netflix's own website doesn't explain a lot), so I can only guess that whatever the reason for the change it was to help them get better data for their analysis and recommendation algorithms.

Here are some factors that I can see between the two types of rating systems that you can take into consideration:

• Ease of completion - always a big factor when it comes to getting users to contribute feedback. If their motivation is high then chances are they will be willing/happy to do something more difficult.
• Power of discrimination - something to think about based on the accuracy of the information that you want to collect and the result that you want to present.
• Ease of implementation - hopefully there is no real difference between the two, but there might be if you use one and not the other already and there are changes required that affects other parts of the system
• Comparison to competitors - something to consider if you want to benchmark or compare to similar apps or the competition out there.
• Use of the data - this will require more consideration because you have to work out what you want to do with the data and how it will be analysed and interpreted to get the insights that you want. The more complex and detailed the question you ask, the more sophisticated the analysis will need to be.

Hope that helps you with understanding that there is no 'correct' answer here, just what is going to work better for your situation given certain constraints/factors.