16

Quoting myself from this other question, there are users that rate us with 1 star when the review is "Good product", as well as the other way around, rating us 5 stars for "Bad product" reviews, sometimes.

It's incredibly strange from my point of view, since I've always seen the star system the way it's intended (the more filled stars, the better), but it's quite a trend that some users don't see it the same way.

I know that for numeric scales it may happen, somehow, in Germany (though I've seen both cases, where 1 means "the best" or "the worst"), but I didn't know it could happen in this case.

So, after this relatively personal explanation, could it be that stars are not a good rating system?

Or is it just that they are not taking it that seriously?

  • 2
    If stars don't work, go with smiley faces: :C :( :| :) :D... A real world example is Changi Airport in Singapore, where restrooms and checkpoints have touchscreen displays with these smileys for you to rate your experience. – ADTC Nov 18 '14 at 17:37
  • Seems like there are two distinct questions to ask on this topic, rather than merging them as one (as some of the answers seem to do). (1) Are star ratings a good system for end users to rate an entity? (2) Are star rankings systems - irrespective of who or what algorithm generated a ranking - a good way for end users to understand an entity's high/good or low/bad ranking? – mg1075 Aug 24 '17 at 3:35
13

I wouldn't say so because it is simply too confusing. As everyone else has posted, star ratings are subjective and some may see 3/5 being worse that others.

When you want users rating on something, it is best to simplify it even further so the user can intuitively know whats going on.

That is why Youtube switched over from stars to liking/disliking a video which had backlash in the beginning but it gradually improved over time and it is a better representation of the video by the community. You see this with many other popular sites as well.

See the following from Margaret Gould Stewart, UX Lead at YouTube (previously Facebook and Google):

"Now, years ago, when I was working at YouTube, we were looking for ways to encourage more people to rate videos, and it was interesting because when we looked into the data, we found that almost everyone was exclusively using the highest five-star rating, a handful of people were using the lowest one-star, and virtually no one was using two, three or four stars. So we decided to simplify into an up-down kind of voting binary model. It's going to be much easier for people to engage with.

But people were very attached to the five-star rating system. Video creators really loved their ratings. Millions and millions of people were accustomed to the old design. So in order to help people prepare themselves for change and acclimate to the new design more quickly, we actually published the data graph sharing with the community the rationale for what we were going to do, and it even engaged the larger industry in a conversation, which resulted in my favorite TechCrunch headline of all time: 'YouTube Comes to a 5-Star Realization: Its Ratings Are Useless.'"

http://www.ted.com/talks/margaret_gould_stewart_how_giant_websites_design_for_you_and_a_billion_others_too/transcript?language=en

  • Welcome to the site, @Jason. Do you have any evidence to support the claims you make about YouTube's switch. Please cite sources. – Graham Herrli Nov 19 '14 at 4:35
  • Subjectivity is a primary reason I dislike any-star ratings. I often wonder if I've screwed up my Netflix recommendations because I think 3/5 is a perfectly respectable movie, while Netflix thinks different. My wife also asks me to rate new recipes, which always ends in us explaining why 3/5 is (or is not) a rating worthy to give for a meal we'd make again. Always think about that when I click a star. :P – Evil Closet Monkey Nov 19 '14 at 23:55
  • @EvilClosetMonkey thanks a lot for the source. I will try to back up statements in the future with sources. – Jason Nov 20 '14 at 7:12
  • Props for the source link goes to @kbwatts. I just put it in a quote block. :) – Evil Closet Monkey Nov 20 '14 at 16:16
  • 1
    If indeed big data shows this, why Amazin, which is huge never changed their rating system? are they blind to the flaws which you described? does the system works better for products then for videos? I'm sure they did enough A/B testing and internal testing to know what works and what doesn't. unless, they didn't... – vsync Oct 30 '15 at 22:43
7

You have several possible visualizations of star ratings 0–5 or 1–5 (worst–best). Some give users more clues how to interpret them than others, some only work for a specific writing direction (mostly left-to-right). Most star rating systems are iconic fixed-step gauges, often they’re interactive.

Rating = number of visible or lit stars

  • stars with arbitrary color hint
    Rating may be accompanied by color change (see other questions) of the stars themselves or their canvas, but that’s easily arbitrary hence not really helpful.
  • fixed width star area and background color
    Rating may be inside a fixed-width box that obviously fits five to intuitively show the maximum number of stars awarded.
  • bigger = better
    Each filled star may be getting bigger than the one before to show direction unambiguously, but the maximum may be unknown. (This can be used in less obvious ways than in the picture.)
  • stars and placeholdersshining stars
    Shining stars besides inactive stars (black / gray / white) or other placeholders like bullets also clearly show the maximum, except when direction is unclear or when stars in the middle may be unlit (like biathlon targets).
  • plus instead of half star
    Semi-filled stars or an additional symbol may boost the number of steps from 5 (or 6) to around 10.
  • military ranks, partially employing clustered stars
    Non-linear alignment may be conventional some places like the military, but it doesn’t clearly show the maximum number of stars, cf. Dice Sides ⚀⚁⚂⚃⚄⚅.

Related

  • jags rating with five-pointed starHarvey Balls
    Moon Rating (radial gauge) using stars: rating = filled jags with 0 = hollow star and 5 = filled star (cf. Harvey Balls), not to be confused with half-star rating; hypothetic variant: Jags Rating with five-pointed star somewhere in the middle •▴✦★✶✷.
  • various colored stars
    Badge Rating using stars, e.g. white, red, blue, silver, gold star (for instance at E-Bay as another answer shows)
  • Heart Rating (♡♥) or Like using a single star, also used as Add Favorite alias Bookmark
  • Up and Down Votes (2 choices ⬆︎⬇︎ ⊕⊖), Thumb Rating (2 or 3 choices 👍👎) or Smiley Rating (2 or more choices ☺︎☹) using a hollow ☆ and a filled star ★ or differently colored stars – never seen that one used, though
  • Australian energy efficiency label
    Semicircular Gauge Rating with stars as extra indicator
  • Symbolic Counter using stars, e.g. on international football jerseys (may be combined with different color or different symbol for more than five victories)
  • Is the first one a snowflake or the fourth star?
    Increased maximum (like 11 on volume knob, AAA+ credit rating or 7-star hotels) only works once or only for a short time as a publicity stunt.

Some sites and apps even try to show two different ratings with one set of stars: yours and the public one.

achromatic stars
If you’re using any other number than 5 (as also found in many Likert scales) or black stars with white placeholders on gray background, you’re asking for trouble, but otherwise stars can do a fine job as rough indicators. It’s questionable, though. whether they’re good at being an input widget, too (see @Jason’s answer detailing the You Tube example). Sorry for the vast answer preceding that concise statement.

Also, XKCD got it right once again.

3

Stars rating systems have been working pretty well in practice from the user sideon giving a general idea of the quality of a product, but we can't expect more than that from them.

The rating is familiar with most of the users because of:

  • Familiarity on the real world: It's used worldwide in hotels as a quality scale.

  • Familiarity on the web: It has been used throughout massive-traffic websites with consistent results (shops sites, movies and series sites,etc).


There are some features to take in count to make them work as good as they can:

  • Gradient from red to green

  • Show description on hover

    enter image description here

And when you display products, get sure of showing the number of votes done to make the rating more meaningful.

enter image description here


Something to avoid: This way of showing reputation (Ebay as an example). Endless and impossible to remember colored stars.

enter image description here


As it is not possible to make some "coherence validation" between the amount of stars selected by the user and their comments, there always be some users that will fail with or without porpoise, but at the end it would tend to be the minority.

I think that the thing about the users' tendency of voting 1 or 5 stars is something to take in count while analyzing this model, but the Like-Dislike approach is too extremist for rating products IMHO. For me a good approach would be a system with 3 states, positive + negative + neutral, because talking about products you could think something is good but "not good enough", or "not as expected" but still useful, and that information will be really useful for the product seller. Maybe 5 stars are too much for the average user to "decide" between them, so they end up falling on the extremes.

  • Ah, right, E-Bay – that was the popular example I was looking for to explain my “badge rating”. – Crissov Nov 18 '14 at 20:55
2

Star rating systems require a lot of cognitive energy.

When Youtube switched from a star-based system to the binary thumbs-up\thumbs-down, their ratings shot up many-fold. Quantifying an emotion at micro-levels (I imagine no one feels too much of anything towards most Youtube videos) is hard.

However, it is not at all foreign to modern humans. Research questionnaires have it, Movie reviews have it, and a lot of online content has it.

I don't know anything about your product, but if people are rating at an acceptable amount - the system is good. The problem is one of clarification.

A research questionnaire would have an explicit "... where 1 is lowest and 5 is highest." A website needs to be more elegant.

I personally feel colors are too ambiguous. Sure, gold, silver and bronze are understandable, but that's just three. Mixing in blue and red with those is a bit strange.

A mouse-over description is a good way to go, but would not be relevant for mobile users. Another possible solution is going for something more linguistic and less numeric. Having the user compare a word or two with their own linguistic interpretation of the emotion is far simpler that rating it on a scale of 1-5.

2

Not really on the question of stars, but on the difference between 1-7 (Likert-like) scales and like/dislike options:

If you use the mathematical average for the 1-7 scales, there's no difference between the case where everyone voted 4 and the case where half voted 1 and half voted 7.

So the 1-7 scales only benefit you if you also use (not necessarily show to the users) the statistical variance (which is different between the above examples, showing there is no consensus in the second example).

I've never seen the variance visualized with a star system - in statistical papers, I think it is shown as a bar around the average value. The length of the bar shows how far on the scale ratings are scattered. For example, this one here.

1

My personal experience with the most complex rating system "ebay".

I have bought and sold many items on this site. What I found is that about 77% give feedback and each feedback is worth 1 point on the scale. Positive feedback is mostly the speed of your buying / shipping and how fast one gets or receives an item. Negative feedback mostly comes from items received late, broken, wrong item, and believe it or not, people who can't read the item description and buy it in error. Listing something for parts or not working with big red letters stating so doesn't mean that it will be read by the buyer so if it sells close to the same price as a working item, a painful week is what's coming. ebay will not remove negative feedback no matter how wrong it is further damaging the feedback system as a whole. I do like the color star scale but it should be for how many transactions a member made with thumbs up and thumbs down count that has comments if one wishes to leave one.

  • So with your experience, are stars good or bad? – Benny Skogberg Oct 25 '16 at 7:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.