Are ratings relevant, and what makes a rating system good?

Question

I was thinking of websites like YouTube, or IMDB, which implement rating systems. On YouTube, a video can be rated by up votes, or down votes, and videos are rated by the ratio of up and down votes. On IMDB, you can rate a movie on a 1 to 10 scale, and the rating is an average.

My first question is how relevant are these ratings to show how good a material is (whether it's a book, or a movie, or a video, or a song)? I think most rating systems suffer from being very subjective. Here are some examples in which the ratings don't reflect at all the quality of the material:

• The message of the material is somewhat controversial. Consumers will give a higher rating if they agree with the message, or a lower rating if they don't.
• The material is not targeted to a particular consumer. For example, someone with no knowledge in a specific domain consumes a material that is targeted to someone who already has some knowledge. As a result, that consumer will give a low rating because he didn't understand it.
• People have different tastes. Someone may like a genre of music, and dislike another genre. Same is with movies. This creates a big problem with the rating because it no longer tells how good that material is, but instead, how popular it is, or how many of the people who consumed it liked it. This is subjective.

So the second question is: what makes a rating system good? Does objectivity make it good, and is it possible to have a rating system that is objective, and truly reflects the quality of the rated material?

Answer 1

I'd like to point out that we are currently operating in a website that uses a rating system and does it quite successfully. There's also a component of gamification, but the rating system itself is quite effective ensuring quality content (mostly because of user discipline and moderators).

You are completely right about the subjectivity in the ratings, especially in Youtube, IMDB and alike and there's nothing wrong with that. You just have to be aware what dimension does the rating measure. Typically it is a popularity contest.

Now, if you'd like to measure your content along other dimensions, like in IMBD context you should explicitly introduce this dimension and let users rate it. Lets say you could ad rating for Screenplay, Plot, Actors performance, Sound quality, Special effects etc.

So if you have only one dimension, the rating reverts to popularity contest by default, unless you have a strong community (and moderators) that are able to establish better rating standards.