Does usage of thumbs up/down ratings as social proof actually provide the right feedback?

Question

On reading the news about Netflix switching to the Facebook 'thumbs up/down' system of rating instead of the more traditional star scale, and seeing that it was on the basis of an overwhelming increase in people providing feedback, I wondered if was really an indication of trends changing in how people provide feedback.

It seems logical that if you are comparing the amount of feedback or engagement between the five start rating with the thumbs up/down rating, it is not hard to see that more people will find it easier to provide feedback because mentally it is an easier decision to make. The general argument with providing a finer degree of measure (as with the debate on a five or seven point scale) is that there are subtle differences you can detect that gets buried because you are not measuring those differences like the variation in the number of star ratings given to create the average. And on the other end of the scale, a more generic measure (such as the NPS score) is suppose to capture the change without giving you an understanding of underlying changes, but prevents you from reading too much into variation that may not reflect the overall change in the trend.

But if you get back to the point of whether a five point scale is a suitable measure of the quality of a movie, most people would say that it is inadequate, simply because a single five point scale doesn't capture all the different aspects of a movie that people factor in to provide a rating, and we don't necessarily compare movies objectively either. This is why in the case of hotel ratings, you'll find that some sites have increased their rating to incorporate multiple categories. This has allowed users to get a better understanding of which aspects of the hotel is rated and how it is weighted, so if they are willing to sacrifice price for convenience then they are more confident of making that choice.

It seems to me that if you wanted a more accurate system of matching movie choices with users then you would take the approach of creating a more detailed rating/review system. Even websites like Rotten Tomato has a reasonably sophisticated system that is not just based on % ratings. However, the fact that Netflix has taken the other approach seems to suggest that because people are viewing more things these days, it means that they don't necessarily spend as much time thinking about or evaluating the things that they watch. If you compare this to websites reviewing books, perhaps simply a thumbs up/down review wouldn't be adequate.

My question is whether the decision by Netflix to change the rating system actually serves the purpose of providing a more accurate review and matching of movie viewing preferences, since you generally have to try and measure more to understand things better?

Answer 1

Let me stay aside from the purpose of this change. There is some business decision behind it, for sure. But regarding the scale itself I am sure it is no good for the accuracy and telling good from better.

Thumb up/down does not allow that because it lacks proper resolution. It does not allow to register "good" and "better" at the very beginning. It is a simple, -1/1 measure. It is, indeed, anchored in the world of quality, but in fact being mostly qualitative.

At the same time, I strongly believe that a 1-5 (or 0-5) scale is just as accurate for that as it should be. It allows me to say a lot about a movie that -/+ does not:

• "Dude, that was a total massacre, don't even think about seeing this film!"
• "It was bad, but not that bad."
• "Well, just another movie like a lot of them. A good one to kill time."
• "Wow, it was actually quite good!"
• "That was a masterpiece!"

As a result, thumbs down/up system throws all the masterpieces and quite good movies into the same bin. Moreover, putting more weight on the number of voters, it makes those exquisite niche movies be outscored by that family comedy that just hit the screens. It dangerously pushes the measure from "how good" towards "how many". And this is bad for both rating and sensing the quality.

EDIT:

So, going further. I think that introducing this kind of measure deteriorates the matching significantly.

• For a 1-5 rating even one or two votes can shape my opinion somehow, and even if I do not know those users' preferences, I would give this movie a try.

• With a thumb up/down mechanism, I need to see e.g. a hundred of them to build up my interest.

In other words, I will get hooked on what is popular, not on what is good. And unless "just an average regular normal guy", this will not match my preferences.

Is there a way out of this? Well, there is! Should my profile be put against the profiles of other users, so that the preference match is at the user profile level a thumb up/thumb down mechanism could be just enough. In other words, I believe it would be possible to match, to some extent, a movie with my preferences based on a combination of a simple thumb up/thumb down system and a profile matching system, that would recommend me movies "upthumbed" by people with similar taste. Unfortunately, for that, a quite elaborate user preferences profile should be built first. As I am not a Netflix user, I cannot tell if it is somewhere there below.

This is something that one of our local movie services, Filmweb, once introduced. I took a part in a long survey that tested my preferences which should result in presenting better suggestions to me. It is a pity I have never tried to use it after that, though, but I still think it could be a good idea. But even then, it is more about recommendations, not rating.